> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Quick Start

> Build your first scraper in 5 minutes with scrapai CLI

## Prerequisites

Before you begin, ensure you have:

* Python 3.9 or higher
* Git
* Terminal access

<Note>
  scrapai works on **Linux**, **macOS**, and **Windows**. The setup process is identical across all platforms.
</Note>

## Installation

<Steps>
  <Step title="Clone the repository">
    ```bash theme={null}
    git clone https://github.com/discourselab/scrapai-cli.git
    cd scrapai-cli
    ```
  </Step>

  <Step title="Run setup">
    ```bash theme={null}
    ./scrapai setup
    ```

    This sets up your environment, installs dependencies, and initializes the database.

    <Tip>
      On **Windows**, use `scrapai setup` instead of `./scrapai setup`.
    </Tip>

    <Warning>
      On **Linux**, if Chromium fails to launch, install system dependencies:

      ```bash theme={null}
      sudo .venv/bin/python -m playwright install-deps chromium
      ```
    </Warning>
  </Step>

  <Step title="Verify installation">
    ```bash theme={null}
    ./scrapai verify
    ```

    You should see:

    ```
    ✅ Virtual environment exists
    ✅ Core dependencies installed
    ✅ Database initialized
    🎉 Environment is ready!
    ```
  </Step>
</Steps>

## Your First Scraper

Let's import and run a pre-built spider for BBC News.

<Steps>
  <Step title="Import the spider">
    scrapai includes example spiders in the `templates/` directory. Let's import the BBC News spider:

    ```bash theme={null}
    ./scrapai spiders import templates/news/bbc_co_uk/analysis/final_spider.json --project news
    ```
  </Step>

  <Step title="Run a test crawl">
    Run the spider in test mode (limits to 5 items):

    ```bash theme={null}
    ./scrapai crawl bbc_co_uk --project news --limit 5
    ```

    <Tip>
      Test mode (`--limit`) stores data in the database for inspection. Production mode (no limit) exports to timestamped JSONL files.
    </Tip>

    You'll see Scrapy crawling in action:

    ```
    2026-02-28 14:30:12 [scrapy.core.engine] INFO: Spider opened
    2026-02-28 14:30:13 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.bbc.co.uk/news/articles/...>
    {'title': 'Breaking news story title', 'content': '...', 'author': 'BBC News', ...}
    ```
  </Step>

  <Step title="View the results">
    Inspect the scraped data:

    ```bash theme={null}
    ./scrapai show bbc_co_uk --project news
    ```
  </Step>

  <Step title="Export the data">
    Export to your preferred format:

    <CodeGroup>
      ```bash CSV theme={null}
      ./scrapai export bbc_co_uk --project news --format csv
      ```

      ```bash JSON theme={null}
      ./scrapai export bbc_co_uk --project news --format json
      ```

      ```bash JSONL theme={null}
      ./scrapai export bbc_co_uk --project news --format jsonl
      ```

      ```bash Parquet theme={null}
      ./scrapai export bbc_co_uk --project news --format parquet
      ```
    </CodeGroup>

    Exports are saved to the `data/` directory with timestamps.
  </Step>
</Steps>

## Explore More Examples

scrapai includes several ready-to-use spider templates:

<CardGroup cols={2}>
  <Card title="E-Commerce" icon="cart-shopping">
    ```bash theme={null}
    ./scrapai spiders import templates/ecommerce/amazon_co_uk_mac_accessories/analysis/final_spider.json --project shop
    ```

    Scrapes product listings with prices, ratings, and descriptions
  </Card>

  <Card title="Forums" icon="comments">
    ```bash theme={null}
    ./scrapai spiders import templates/forums/news_ycombinator_com/analysis/final_spider.json --project forums
    ```

    Extracts discussion threads, authors, and timestamps
  </Card>

  <Card title="Cloudflare-Protected" icon="shield">
    ```bash theme={null}
    ./scrapai spiders import templates/cloudflare/thefga_org/analysis/final_spider.json --project research
    ```

    Demonstrates Cloudflare bypass with cookie caching
  </Card>

  <Card title="Real Estate" icon="house">
    ```bash theme={null}
    ./scrapai spiders import templates/spider-realestate.json --project housing
    ```

    Property listings with custom field extractors
  </Card>
</CardGroup>

## Using with AI Agents

scrapai is designed to work with AI coding agents like Claude Code. Instead of manually writing JSON configs, you describe what you want in plain English:

```bash theme={null}
claude
```

```
You: "Add https://techcrunch.com to my news project"
Agent: [Analyzes site, generates rules, tests extraction, deploys spider]

You: "Crawl all spiders in my news project and export to CSV"
Agent: [Executes crawls, exports data]
```

<Tip>
  The `./scrapai setup` command automatically configures Claude Code permissions to prevent the agent from modifying framework code—it can only write JSON configs and run CLI commands.
</Tip>

## Production Crawling

Run a full crawl by omitting `--limit`:

```bash theme={null}
./scrapai crawl bbc_co_uk --project news
```

A full crawl auto-detaches into **Pueue**, so it keeps running after you disconnect from SSH. scrapai queues the task, prints its Pueue task ID, and returns immediately:

```
Production crawl 'bbc_co_uk' queued in Pueue (task 3); survives SSH disconnect.
  progress: pueue log 3   all: pueue status   stop: pueue kill 3
```

<Note>
  Detached crawls require Pueue. If it isn't installed, scrapai stops and tells you to install it (or test with `--limit N`).
</Note>

Check progress at any time with `crawl-status`:

```bash theme={null}
./scrapai crawl-status bbc_co_uk --project news
```

This joins each crawl's Pueue run state (running, queued, done, ...) with its output file, showing items downloaded and how many have extracted content. Omit the spider name to see every crawl in the project.

Production crawls export to timestamped JSONL, resume from a checkpoint if interrupted, and skip already-seen URLs via DeltaFetch.

<Warning>
  Production crawls can run for hours or days. Use `--limit` for testing first.
</Warning>

## Next Steps

<CardGroup cols={2}>
  <Card title="Installation Guide" icon="download" href="/installation">
    Detailed installation instructions for all platforms
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/cli/overview">
    Complete command reference
  </Card>

  <Card title="Configuration" icon="gear" href="/configuration/environment">
    Configure proxies, databases, and S3 storage
  </Card>
</CardGroup>