> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Parallel Crawling

> Run many production crawls concurrently and detached with Pueue

Run a whole fleet of spiders at once. scrapai submits each production crawl to [Pueue](https://github.com/Nukesor/pueue), a background command queue that runs several crawls in parallel, keeps them alive after you disconnect, and reports them all through one status view.

<Note>
  Parallel crawling reuses the same mechanism as a single production crawl: any `./scrapai crawl` **without** `--limit` is handed to Pueue. There is no separate parallel command to learn — you submit more crawls, and Pueue runs them concurrently. See [Crawl Commands](/cli/crawl) for the single-crawl details.
</Note>

## How it works

When you run a full crawl, scrapai adds it to Pueue with a label of the form `scrapai:<project>:<spider>`:

```bash theme={null}
./scrapai crawl bbc_co_uk --project news
# Production crawl 'bbc_co_uk' queued in Pueue (task 42); survives SSH disconnect.
#   progress: pueue log 42   all: pueue status   stop: pueue kill 42
```

Pueue runs queued tasks up to its **parallelism limit** and queues the rest. Submit many crawls and they fan out automatically — no resource math to do by hand.

## Setup

Install Pueue and start its daemon once per machine (this is the same setup used for any detached crawl):

<CodeGroup>
  ```bash macOS theme={null}
  brew install pueue
  ```

  ```bash Linux (static binary — any distro, no root) theme={null}
  # Grab the latest pueue + pueued from the GitHub releases page.
  # Works on any distro (musl static build); no root needed.
  mkdir -p ~/.local/bin
  base=https://github.com/Nukesor/pueue/releases/latest/download
  curl -L "$base/pueue-x86_64-unknown-linux-musl"  -o ~/.local/bin/pueue
  curl -L "$base/pueued-x86_64-unknown-linux-musl" -o ~/.local/bin/pueued
  chmod +x ~/.local/bin/pueue ~/.local/bin/pueued
  # ensure ~/.local/bin is on PATH (add to ~/.bashrc if not):
  export PATH="$HOME/.local/bin:$PATH"
  ```

  ```bash Linux (distro package) theme={null}
  # Only on newer distros that package pueue
  # (Debian 12+, Ubuntu 23.04+, current Arch). On older
  # releases the package is missing — use the static binary instead.
  sudo apt install pueue      # or: sudo pacman -S pueue
  ```

  ```bash Any (via Rust) theme={null}
  cargo install --locked pueue
  ```
</CodeGroup>

<Note>
  `sudo apt install pueue` only works on newer distributions (Debian 12+, Ubuntu 23.04+). On Ubuntu 22.04 LTS and earlier the package isn't in the repositories — use the **static binary** tab, which works on any distro without root. On ARM servers swap `x86_64` for `aarch64` in the binary names.
</Note>

Start the daemon so the queue survives logout and reboot:

```bash theme={null}
systemctl --user enable --now pueued    # start now + on every boot, auto-restart on crash
loginctl enable-linger $USER            # keep it running with no active login
pueue status                            # verify — prints an empty queue table
```

<Tip>
  Quick, non-persistent alternative: `pueued -d` (survives disconnect but not reboot).
</Tip>

## Set how many run at once

By default Pueue runs one task at a time. Raise the limit to crawl several sites concurrently:

```bash theme={null}
pueue parallel 8      # run up to 8 crawls at once
```

Pick a number your machine can handle. Cloudflare/browser-enabled spiders (`--browser`) use far more memory than plain HTTP crawls, so keep the limit lower when the fleet is browser-heavy.

## Monitor the fleet

Check every detached crawl at a glance — run state plus how much each has downloaded:

```bash theme={null}
./scrapai crawl-status --project news
# spider          project   state     downloaded    with-content   start           end  last-item
# clarin_com      news      running        1,240    1,198 (97%)     13:53 29-06-26  -    4s
# bbc_co_uk       news      queued             -             -      -               -    -
```

* **state** — Pueue's run state: `running`, `queued`, `paused`, `done`, `killed`, `failed`.
* **with-content** — share of downloaded items that carry content text (PDF/links-only items are excluded — they carry no content by design).
* **last-item** — time since the crawl last wrote an item, so a running crawl that has stalled stands out.

Raw Pueue works too:

```bash theme={null}
pueue status            # all tasks
pueue log <task-id>     # output of one crawl
pueue kill <task-id>    # stop one crawl
pueue kill --all        # stop everything
```

## Failure handling

Each crawl is an independent Pueue task, so one spider failing doesn't stop the others. A failed crawl shows as `failed` in `crawl-status` / `pueue status`; inspect it with `pueue log <task-id>` and resubmit with the same `./scrapai crawl` command once fixed.

## Scheduling recurring fleets

Pueue handles *running* crawls; use system cron (or a systemd timer) to *trigger* them on a schedule. Queue a whole project from one cron line — each `./scrapai crawl` hands off to Pueue and returns immediately:

```bash theme={null}
# crontab -e — full news fleet every day at 2am
0 2 * * * cd /path/to/scrapai-cli && for s in $(./scrapai spiders list --project news | grep '•' | awk '{print $2}'); do ./scrapai crawl "$s" --project news; done >> logs/crawl.log 2>&1
```

Because each crawl is detached under Pueue, the cron job returns immediately after queuing — the crawls keep running in the background.

## See Also

<CardGroup cols={2}>
  <Card title="Crawl Commands" icon="spider" href="/cli/crawl">
    Single crawls, flags, and crawl-status details
  </Card>

  <Card title="Checkpoint Resume" icon="pause" href="/guides/checkpoint-resume">
    Pause and resume long crawls
  </Card>
</CardGroup>