Skip to main content

​
Prerequisites

Before you begin, ensure you have:
  • Python 3.9 or higher
  • Git
  • Terminal access
scrapai works on Linux, macOS, and Windows. The setup process is identical across all platforms.

​
Installation

1

Clone the repository

2

Run setup

This sets up your environment, installs dependencies, and initializes the database.
On Windows, use scrapai setup instead of ./scrapai setup.
On Linux, if Chromium fails to launch, install system dependencies:
3

Verify installation

You should see:

​
Your First Scraper

Let’s import and run a pre-built spider for BBC News.
1

Import the spider

scrapai includes example spiders in the templates/ directory. Let’s import the BBC News spider:
2

Run a test crawl

Run the spider in test mode (limits to 5 items):
Test mode (--limit) stores data in the database for inspection. Production mode (no limit) exports to timestamped JSONL files.
You’ll see Scrapy crawling in action:
3

View the results

Inspect the scraped data:
4

Export the data

Export to your preferred format:
Exports are saved to the data/ directory with timestamps.

​
Explore More Examples

scrapai includes several ready-to-use spider templates:

E-Commerce

Scrapes product listings with prices, ratings, and descriptions

Forums

Extracts discussion threads, authors, and timestamps

Cloudflare-Protected

Demonstrates Cloudflare bypass with cookie caching

Real Estate

Property listings with custom field extractors

​
Using with AI Agents

scrapai is designed to work with AI coding agents like Claude Code. Instead of manually writing JSON configs, you describe what you want in plain English:
The ./scrapai setup command automatically configures Claude Code permissions to prevent the agent from modifying framework code—it can only write JSON configs and run CLI commands.

​
Production Crawling

Run a full crawl by omitting --limit:
A full crawl auto-detaches into Pueue, so it keeps running after you disconnect from SSH. scrapai queues the task, prints its Pueue task ID, and returns immediately:
Detached crawls require Pueue. If it isn’t installed, scrapai stops and tells you to install it (or test with --limit N).
Check progress at any time with crawl-status:
This joins each crawl’s Pueue run state (running, queued, done, …) with its output file, showing items downloaded and how many have extracted content. Omit the spider name to see every crawl in the project. Production crawls export to timestamped JSONL, resume from a checkpoint if interrupted, and skip already-seen URLs via DeltaFetch.
Production crawls can run for hours or days. Use --limit for testing first.

​
Next Steps

Installation Guide

Detailed installation instructions for all platforms

CLI Reference

Complete command reference

Configuration

Configure proxies, databases, and S3 storage