Skip to main content
Run a whole fleet of spiders at once. scrapai submits each production crawl to Pueue, a background command queue that runs several crawls in parallel, keeps them alive after you disconnect, and reports them all through one status view.
Parallel crawling reuses the same mechanism as a single production crawl: any ./scrapai crawl without --limit is handed to Pueue. There is no separate parallel command to learn — you submit more crawls, and Pueue runs them concurrently. See Crawl Commands for the single-crawl details.

​
How it works

When you run a full crawl, scrapai adds it to Pueue with a label of the form scrapai:<project>:<spider>:
Pueue runs queued tasks up to its parallelism limit and queues the rest. Submit many crawls and they fan out automatically — no resource math to do by hand.

​
Setup

Install Pueue and start its daemon once per machine (this is the same setup used for any detached crawl):
sudo apt install pueue only works on newer distributions (Debian 12+, Ubuntu 23.04+). On Ubuntu 22.04 LTS and earlier the package isn’t in the repositories — use the static binary tab, which works on any distro without root. On ARM servers swap x86_64 for aarch64 in the binary names.
Start the daemon so the queue survives logout and reboot:
Quick, non-persistent alternative: pueued -d (survives disconnect but not reboot).

​
Set how many run at once

By default Pueue runs one task at a time. Raise the limit to crawl several sites concurrently:
Pick a number your machine can handle. Cloudflare/browser-enabled spiders (--browser) use far more memory than plain HTTP crawls, so keep the limit lower when the fleet is browser-heavy.

​
Monitor the fleet

Check every detached crawl at a glance — run state plus how much each has downloaded:
  • state — Pueue’s run state: running, queued, paused, done, killed, failed.
  • with-content — share of downloaded items that carry content text (PDF/links-only items are excluded — they carry no content by design).
  • last-item — time since the crawl last wrote an item, so a running crawl that has stalled stands out.
Raw Pueue works too:

​
Failure handling

Each crawl is an independent Pueue task, so one spider failing doesn’t stop the others. A failed crawl shows as failed in crawl-status / pueue status; inspect it with pueue log <task-id> and resubmit with the same ./scrapai crawl command once fixed.

​
Scheduling recurring fleets

Pueue handles running crawls; use system cron (or a systemd timer) to trigger them on a schedule. Queue a whole project from one cron line — each ./scrapai crawl hands off to Pueue and returns immediately:
Because each crawl is detached under Pueue, the cron job returns immediately after queuing — the crawls keep running in the background.

​
See Also

Crawl Commands

Single crawls, flags, and crawl-status details

Checkpoint Resume

Pause and resume long crawls