Skip to main content
scrapai automatically enables checkpoint support for production crawls, allowing you to pause long-running crawls and resume them later without losing progress.

​
How It Works

1

Automatic for production crawls

Checkpoint is automatically enabled when running production crawls (no --limit flag)
2

Press Ctrl+C to pause

Checkpoint saved automatically when interrupted
3

Run same command to resume

Automatically detects checkpoint and resumes from where you left off
4

Automatic cleanup on success

Checkpoint deleted automatically on successful completion
Test crawls (with --limit) do not use checkpoints since they’re short-running.

​
What Gets Saved

Scrapy’s JOBDIR feature saves:
  1. Pending requests - All URLs waiting to be crawled
  2. Duplicates filter - URLs already visited (prevents re-crawling)
  3. Spider state - Any custom state stored in spider.state dict

​
Usage

​
Production Crawl with Checkpoint

Console output:
Pause the crawl:
Resume later:

​
Test Crawl (No Checkpoint)

Console output:

​
Checkpoint Storage

Checkpoints are stored in your DATA_DIR:
Example directory structure:

​
Cleanup

Manual cleanup:

​
Complete Example

1

Start production crawl

Output:
2

Pause crawl (Ctrl+C)

Output:
3

Check checkpoint exists

Output:
4

Resume crawl

Output:

​
Limitations

Request callbacks must be spider methods (Scrapy limitation):
✅ scrapai spiders already compatible: Our database spiders use spider methods (self.parse), so checkpoints work out of the box!
Other limitations:
  • Cookie expiration: If you wait too long to resume (days/weeks), cookies may expire and requests may fail. Resume within a reasonable timeframe (hours/days, not weeks).
  • Multiple runs: Each spider should have only one checkpoint at a time. Don’t run the same spider concurrently while a checkpoint exists.
  • Proxy type changes: If you change --proxy-type when resuming, the checkpoint is automatically cleared (see below).

​
Proxy Type Changes (Expert-in-the-Loop)

If you change --proxy-type when resuming, the checkpoint is automatically cleared and crawl starts fresh.
Example scenario:
1

Start crawl with auto mode

Uses datacenter proxies (auto mode default)
2

Datacenter fails, pause crawl

3

Resume with residential proxy

Output:
Why checkpoint is cleared:
  • Ensures blocked URLs are retried with new proxy type
  • Prevents Scrapy’s dupefilter from skipping already-seen failed URLs

​
When Checkpoints Are Useful

✅ Long-running crawls (hours/days) Resume if interrupted✅ Unstable connections Resume after network failures✅ System maintenance Pause before server restart, resume after✅ Resource management Pause during high-load periods, resume later

​
Technical Details

Built on Scrapy’s JOBDIR:
  • Uses Scrapy’s native pause/resume feature (not custom implementation)
  • Checkpoint files are pickle-serialized Scrapy objects
  • Atomic writes prevent checkpoint corruption
  • Compatible with all Scrapy spiders
Directory per spider:
  • Each spider gets its own checkpoint directory
  • Prevents conflicts between spiders
  • Clean separation of state
Smart cleanup:
  • Exit code 0 (success) → cleanup checkpoint
  • Exit code != 0 (error/Ctrl+C) → keep checkpoint for resume

​
Troubleshooting

​
Checkpoint Not Resuming

1

Check if checkpoint exists

If directory doesn’t exist:
  • Checkpoint was cleaned up (successful completion)
  • Or never created (test mode with --limit)
2

Check for proxy type change

Changing --proxy-type clears checkpoint automatically

​
Start Fresh (Discard Checkpoint)

​
Checkpoint from Old Spider Version

If you updated spider rules/selectors significantly, old checkpoint may be incompatible.
Solution:

​
Checkpoint Files Too Large

Check size:
Large checkpoints indicate:
  • Many pending URLs (normal for large crawls)
  • Consider crawling in smaller batches
  • Or use incremental crawling (DeltaFetch)

Incremental Crawling

Skip unchanged pages on subsequent crawls

Queue Processing

Batch process multiple websites

Proxy Escalation

Smart proxy usage with cost control