> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Commands

> Run test crawls in the foreground and full production crawls as detached Pueue tasks

Crawl commands run spiders stored in the database. scrapai has two execution paths, chosen by whether you pass `--limit`:

* **Test crawl** (`--limit N`): bounded, runs in the foreground. Use it to verify a spider before a full run.
* **Production crawl** (no `--limit`): the full crawl. scrapai hands it to **Pueue** so it keeps running after you disconnect from SSH.

<Note>
  Pueue is an operational dependency for production crawls. If it is not installed, a `crawl` without `--limit` exits with an error and tells you to install Pueue (see the README, "Long-running crawls") or to test with `--limit N`.
</Note>

## crawl

Run a spider by name.

```bash theme={null}
./scrapai crawl <spider> --project <name> [options]
```

### Arguments & Options

<ParamField path="spider" type="string" required>
  Spider name (from the database).
</ParamField>

<ParamField path="--project" type="string" required>
  Project name containing the spider.
</ParamField>

<ParamField path="--output, -o" type="string">
  Output file path. Defaults to a date-based file in `data/<project>/<spider>/crawls/`.
</ParamField>

<ParamField path="--limit, -l" type="integer">
  Limit number of items. When set, runs a **test crawl** in the foreground.
</ParamField>

<ParamField path="--timeout, -t" type="integer">
  Max runtime in seconds. Sets `CLOSESPIDER_TIMEOUT` for a graceful stop.
</ParamField>

<ParamField path="--proxy-type" type="string" default="auto">
  Proxy to use: `auto` (smart escalation), `none`, or any named profile configured in `.env` (e.g. `datacenter`, `residential`, `isp`, `mobile`).
</ParamField>

<ParamField path="--browser" type="boolean">
  Use a browser for JS-rendered sites and Cloudflare bypass. Sets `CLOUDFLARE_ENABLED=True`.
</ParamField>

<ParamField path="--scrapy-args" type="string">
  Additional Scrapy arguments, e.g. `"-s SETTING=value -L DEBUG"`.
</ParamField>

<ParamField path="--reset-deltafetch" type="boolean">
  Clear the DeltaFetch cache to re-crawl all URLs. Also clears the checkpoint.
</ParamField>

<ParamField path="--save-html" type="boolean">
  Save raw HTML in the output (`INCLUDE_HTML_IN_OUTPUT=True`). Makes files larger.
</ParamField>

<Note>
  There is an internal `--detached` flag used only when Pueue re-runs the command inside its worker. It is hidden and not meant for direct use.
</Note>

### Test Crawl (with --limit)

```bash theme={null}
./scrapai crawl bbc_co_uk --project news --limit 5
```

**Behavior:**

* Runs in the foreground (not queued to Pueue).
* Stops after N items (`CLOSESPIDER_ITEMCOUNT`).
* Saves to the database. Verify with `./scrapai show bbc_co_uk`.
* Pass `-o <file>` to also write results to a file.

**Output:**

```bash theme={null}
🚀 Running DB spider: bbc_co_uk
🔄 Proxy mode: auto (smart escalation with expert-in-the-loop)
🧪 Test mode: Saving to database (limit: 5 items)
   Use './scrapai show bbc_co_uk' to verify results

[Scrapy crawl output...]

2026-02-28 15:30:42 [scrapy.core.engine] INFO: Spider closed (closespider_itemcount)
```

<Tip>
  Run a test crawl to verify spider configuration before launching a full production crawl.
</Tip>

### Production Crawl (no limit)

```bash theme={null}
./scrapai crawl bbc_co_uk --project news
```

**Behavior:**

* Queued to Pueue and run detached, so it survives an SSH disconnect.
* Exports to a date-based JSONL: `data/news/bbc_co_uk/crawls/crawl_28022026.jsonl` (one file per day; same-day re-runs append).
* Full HTML content is included only with `--save-html`.
* Checkpoint enabled — see [Checkpoint Resume](#checkpoint-pauseresume).
* Uploaded to S3 after a successful run, if S3 is configured.

**Output:**

```bash theme={null}
Production crawl 'bbc_co_uk' queued in Pueue (task 42); survives SSH disconnect.
  progress: pueue log 42   all: pueue status   stop: pueue kill 42
```

Use `crawl-status` (below) for a higher-level view of progress across spiders, or the raw Pueue commands shown above.

<Warning>
  A production `crawl` requires Pueue. If it is not installed:

  ```bash theme={null}
  Pueue not installed - needed to run full crawls detached.
  Install it (README: 'Long-running crawls'), or test with --limit N.
  ```
</Warning>

### Checkpoint Pause/Resume

Production crawls set a Scrapy `JOBDIR` checkpoint at `data/<project>/<spider>/checkpoint/`. To resume, run the same command — it picks up where it left off and continues writing to the same day's file.

To stop a detached crawl, kill its Pueue task (`pueue kill <task-id>`); re-running the command resumes from the checkpoint. The checkpoint is removed automatically on successful completion.

<Note>
  scrapai guards two checkpoint edge cases automatically:

  * **Corrupted checkpoint** (dupefilter persisted but the request queue is empty — a [known Scrapy bug](https://github.com/scrapy/scrapy/issues/4106)): the dupefilter is cleared so URLs can be re-discovered.
  * **Proxy type changed** since the last run: the checkpoint is cleared so all URLs are retried with the new proxy.
</Note>

### Proxy Modes

`--proxy-type` is not a fixed list — it accepts `auto`, `none`, or any profile name you have configured in `.env`.

<CodeGroup>
  ```bash auto (default) theme={null}
  ./scrapai crawl myspider --project myproject --proxy-type auto
  ```

  ```bash none theme={null}
  ./scrapai crawl myspider --project myproject --proxy-type none
  ```

  ```bash named profile theme={null}
  ./scrapai crawl myspider --project myproject --proxy-type residential
  ```
</CodeGroup>

* `auto` — smart escalation with expert-in-the-loop (default).
* `none` — direct connections only.
* any other name — an explicit profile, used when blocked.

See [Proxy Configuration](/configuration/proxies) for `.env` setup of named profiles.

<Warning>
  Changing the proxy type between runs clears the checkpoint so all URLs are retried with the new proxy.
</Warning>

### Timeout

```bash theme={null}
./scrapai crawl myspider --project myproject --timeout 7200  # 2 hours
```

Sets `CLOSESPIDER_TIMEOUT` for a graceful stop (finishes in-flight requests, saves the checkpoint).

### Cloudflare / Browser Mode

For sites that need a browser (JS rendering or Cloudflare), use `--browser` or set `CLOUDFLARE_ENABLED: true` in the spider settings:

```bash theme={null}
./scrapai crawl cloudflare_spider --project myproject --browser
```

On headless Linux servers the crawler auto-detects the missing display and wraps the run with `xvfb-run -a`. If Xvfb is not installed, browser mode errors out and tells you to `sudo apt-get install xvfb` (or force headless with `CLOUDFLARE_HEADLESS=true`). See the [Cloudflare Bypass](/guides/cloudflare-bypass) guide.

### Sitemap Spider

For spiders with `USE_SITEMAP: true`, scrapai uses the sitemap spider and crawls from the XML sitemap instead of following links.

## crawl-status

Show each detached crawl's Pueue run state alongside how much it has downloaded. Reads `pueue status --json` and joins it with the latest crawl file per spider.

```bash theme={null}
./scrapai crawl-status [<spider>] [--project <name>]
```

<ParamField path="spider" type="string">
  Optional. Report just this spider (cheaper — only its crawl file is read). Omit to report every spider.
</ParamField>

<ParamField path="--project" type="string">
  Only show crawls in this project.
</ParamField>

**Output:**

```bash theme={null}
  spider      project   state     downloaded   with-content   start           end             last-item

  bbc_co_uk   news      running        1,234    1,180 (96%)    15:30 28-02-26  -               4s
  guardian    news      done           8,902    8,540 (95%)    09:12 27-02-26  14:03 27-02-26  17h
```

**Columns:**

* **state** — Pueue run state: `running`, `queued`, `paused`, `done`, `killed`, or `failed`.
* **downloaded** — items in the latest crawl file.
* **with-content** — items whose extracted `content` is non-empty, with a percentage. The **percentage excludes PDFs** (collected links-only, no content by design), so a healthy crawl isn't dragged down by PDFs.
* **start / end** — Pueue task start and end times (`-` when not yet ended).
* **last-item** — time since the crawl file was last written — a liveness signal for a running crawl.

<Note>
  `crawl-status` only knows about crawls Pueue tracked (detached production crawls, labelled `scrapai:<project>:<spider>`). If Pueue is not installed it reports that there are no detached crawls to show.
</Note>

## crawl-all

Run all **active** spiders in a project, sequentially and in the foreground.

```bash theme={null}
./scrapai crawl-all --project <name> [--limit <N>]
```

<ParamField path="--project" type="string" required>
  Project name.
</ParamField>

<ParamField path="--limit, -l" type="integer">
  Limit items per spider (test crawl for each).
</ParamField>

```bash theme={null}
./scrapai crawl-all --project news --limit 10
```

<Note>
  `crawl-all` runs each spider inline, one after another — it does not queue them to Pueue.
</Note>

## Output Formats

### JSONL (Production)

Each line is a JSON object. `url` and `content` are the fields scrapai reads for status reporting; a spider's own fields (title, html, timestamps, …) are also written per its schema.

```jsonl theme={null}
{"url": "https://bbc.co.uk/news/article-1", "content": "..."}
{"url": "https://bbc.co.uk/news/article-2", "content": "..."}
```

With `--save-html`, the raw HTML is included in each record.

### Database (Test Crawl)

Test crawls save to the `scraped_items` table:

```sql theme={null}
SELECT id, url, title, scraped_at
FROM scraped_items
WHERE spider_id = (SELECT id FROM spiders WHERE name = 'bbc_co_uk');
```

## Troubleshooting

### Spider Not Found

```bash theme={null}
❌ Spider 'myspider' not found in project 'myproject'.
```

Import the spider first:

```bash theme={null}
./scrapai spiders import myspider.json --project myproject
```

### Pueue Not Installed

A production crawl needs Pueue. Install it (README: "Long-running crawls"), or run a bounded test with `--limit N`.

### Checkpoint Corruption

If resume fails, clear the checkpoint and restart:

```bash theme={null}
rm -rf data/<project>/<spider>/checkpoint
./scrapai crawl <spider> --project <project>
```

### Cloudflare Bypass Failed

Browser mode needs a display. On Linux servers install Xvfb (`sudo apt-get install xvfb`), or set `CLOUDFLARE_HEADLESS=true` in the spider settings.

## Next Steps

<CardGroup cols={2}>
  <Card title="View Scraped Data" icon="eye" href="/cli/data">
    Inspect and export crawl results
  </Card>

  <Card title="Cloudflare Bypass" icon="shield" href="/guides/cloudflare-bypass">
    Handle browser-protected sites
  </Card>
</CardGroup>