> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# CLI Overview

> Complete command-line interface reference for scrapai

The scrapai CLI provides a command-line interface for building, managing, and running AI-powered web scrapers. Spiders are stored as JSON configurations in the database.

## Architecture

scrapai uses a project-based organization model:

* **Projects**: Logical groupings of spiders (e.g., `news`, `ecommerce`, `research`)
* **Spiders**: JSON configurations stored in the database
* **Queue**: Database-backed queue for batch processing
* **Data**: Test mode saves to database, production mode exports to JSONL files

## Entry Point

The `scrapai` script automatically activates the virtual environment and delegates to the CLI:

```bash theme={null}
# Linux/macOS
./scrapai <command> [options]

# Windows
scrapai <command> [options]
```

## Command Categories

<CardGroup cols={2}>
  <Card title="Setup & Verification" icon="gear" href="/cli/setup">
    Install dependencies, configure environment, verify setup
  </Card>

  <Card title="Spider Management" icon="spider" href="/cli/spiders">
    List, import, delete, and manage spider configurations
  </Card>

  <Card title="Crawling" icon="globe" href="/cli/crawl">
    Run spiders in test or production mode with checkpoint support
  </Card>

  <Card title="Queue Management" icon="list" href="/cli/queue">
    Add URLs, bulk import, process items in parallel batches
  </Card>

  <Card title="Data Operations" icon="database" href="/cli/data">
    View scraped items, export to CSV/JSON/Parquet
  </Card>

  <Card title="Inspection" icon="magnifying-glass" href="/cli/inspect">
    Analyze websites for scraper development
  </Card>

  <Card title="Database" icon="server" href="/cli/database">
    Migrations, queries, statistics, data transfer
  </Card>

  <Card title="Projects" icon="folder" href="/cli/projects">
    List and manage project configurations
  </Card>
</CardGroup>

## Utility & Diagnostic Commands

Standalone commands for authentication, browser management, selector discovery, and crawl monitoring.

<CardGroup cols={2}>
  <Card title="session" icon="key">
    Capture and manage login sessions — scrapai never types your password. Subcommands: `login`, `check`, `list`, `remove`.
  </Card>

  <Card title="browser" icon="window-maximize">
    Manage the persistent browser service (a warm browser reused across inspect/screenshot calls). Subcommands: `start`, `stop`, `restart`, `status`, `shot`.
  </Card>

  <Card title="analyze" icon="magnifying-glass" href="/cli/inspect">
    Analyze a local HTML file for CSS selector discovery (`--test`, `--find`, `--find-text`).
  </Card>

  <Card title="try" icon="flask">
    Run newspaper and trafilatura against a local HTML file and compare their output.
  </Card>

  <Card title="extract-urls" icon="link" href="/cli/inspect">
    Extract all URLs from an HTML file (`--file`, optional `--output`).
  </Card>

  <Card title="health" icon="heart-pulse" href="/guides/ai-assisted-maintenance">
    Test all spiders in a project and generate a report for broken ones.
  </Card>

  <Card title="crawl-status" icon="gauge">
    Show each crawl's run state and how much it has downloaded (requires Pueue for detached crawls).
  </Card>
</CardGroup>

## Global Conventions

### Project Names

Most commands require a `--project` flag to specify the project context:

```bash theme={null}
./scrapai spiders list --project news
./scrapai crawl bbc_co_uk --project news
```

<Tip>
  Default project is `default` if not specified.
</Tip>

### Output Modes

**Test Mode** (with `--limit`):

* Saves scraped items to database
* Limited number of items
* Use `show` command to view results
* No HTML content stored

**Production Mode** (no limit):

* Exports to timestamped JSONL files in `data/<project>/<spider>/crawls/`
* Includes full HTML content
* Enables checkpoint pause/resume
* Database writes disabled for performance

### File Paths

All data is stored under the `DATA_DIR` configured in `.env` (default: `./data`):

```
data/
├── <project>/
│   ├── <spider>/
│   │   ├── crawls/         # Production JSONL exports
│   │   ├── exports/        # Manual exports (CSV/JSON/Parquet)
│   │   └── checkpoint/     # Pause/resume state
```

## Common Workflows

### Quick Test

Test a spider on 5-10 URLs to verify extraction:

```bash theme={null}
./scrapai crawl myspider --project myproject --limit 5
./scrapai show myspider --project myproject
```

### Production Crawl

Run a full crawl with checkpoint support:

```bash theme={null}
./scrapai crawl myspider --project myproject
# Press Ctrl+C to pause
# Run same command to resume
```

### Batch Processing

Add multiple websites to queue and process:

```bash theme={null}
./scrapai queue bulk urls.csv --project myproject
./scrapai queue list --project myproject
./scrapai queue next --project myproject  # Claim next item
```

### Export Data

Export scraped data in various formats:

```bash theme={null}
./scrapai export myspider --project myproject --format csv
./scrapai export myspider --project myproject --format parquet
```

## Platform Support

* **Linux**: Full support including headless Cloudflare bypass with xvfb
* **macOS**: Full support
* **Windows**: Full support (use `scrapai.bat` or `scrapai` directly)

## Database Support

* **SQLite**: Default, zero configuration
* **PostgreSQL**: Production deployments, atomic queue operations