Skip to main content
ScrapAI automatically enables checkpoint support for production crawls, allowing you to pause long-running crawls and resume them later without losing progress.

​
How It Works

1

Automatic for production crawls

Checkpoint is automatically enabled when running production crawls (no --limit flag)
2

Press Ctrl+C to pause

Checkpoint saved automatically when interrupted
3

Run same command to resume

Automatically detects checkpoint and resumes from where you left off
4

Automatic cleanup on success

Checkpoint deleted automatically on successful completion
Test crawls (with --limit) do not use checkpoints since they’re short-running.

​
What Gets Saved

Scrapy’s JOBDIR feature saves:
  1. Pending requests - All URLs waiting to be crawled
  2. Duplicates filter - URLs already visited (prevents re-crawling)
  3. Spider state - Any custom state stored in spider.state dict

​
Usage

​
Production Crawl with Checkpoint

# Start production crawl (checkpoint auto-enabled)
./scrapai crawl myspider --project myproject
Console output: 
💾 Checkpoint enabled: ./data/myproject/myspider/checkpoint
Pause the crawl: 
# Press Ctrl+C
^C
Resume later: 
# Run same command
./scrapai crawl myspider --project myproject
# Automatically detects checkpoint and resumes

​
Test Crawl (No Checkpoint)

# Test mode - no checkpoint needed (short run)
./scrapai crawl myspider --project myproject --limit 10
Console output: 
🧪 Test mode: Saving to database (limit: 10 items)

​
Checkpoint Storage

Checkpoints are stored in your DATA_DIR: 
DATA_DIR/<project>/<spider>/checkpoint/
Example directory structure: 
./data/myproject/myspider/
├── analysis/        # Phase 1-3 files
├── crawls/          # Production outputs
├── exports/         # Database exports
└── checkpoint/      # Checkpoint state (auto-cleaned on success)

​
Cleanup

Manual cleanup: 
# If you want to discard a checkpoint and start fresh
rm -rf ./data/myproject/myspider/checkpoint/

​
Complete Example

1

Start production crawl

./scrapai crawl techcrunch --project news
Output:
💾 Checkpoint enabled: ./data/news/techcrunch/checkpoint

Crawling: https://techcrunch.com/
Scraped 50 items...
Scraped 100 items...
2

Pause crawl (Ctrl+C)

^C
Output:
Received interrupt signal, shutting down...
Checkpoint saved: 150 items scraped, 237 URLs pending
Run same command to resume from checkpoint
3

Check checkpoint exists

ls -la ./data/news/techcrunch/checkpoint/
Output:
drwxr-xr-x  5 user  staff   160 Feb 24 10:30 .
drwxr-xr-x  7 user  staff   224 Feb 24 10:15 ..
-rw-r--r--  1 user  staff  4096 Feb 24 10:30 requests.queue
-rw-r--r--  1 user  staff  8192 Feb 24 10:30 dupefilter.db
-rw-r--r--  1 user  staff   512 Feb 24 10:30 spider.state
4

Resume crawl

./scrapai crawl techcrunch --project news
Output:
♻️  Resuming from checkpoint: 150 items already scraped, 237 URLs pending

Continuing crawl...
Scraped 160 items...
Scraped 200 items...
Crawl completed successfully!

🧹 Checkpoint cleaned up (crawl completed)

​
Limitations

Request callbacks must be spider methods (Scrapy limitation):
# ✅ Works (spider method)
Request(url, callback=self.parse_article)

# ❌ Won't work (external function)
Request(url, callback=some_external_function)
✅ ScrapAI spiders already compatible: Our database spiders use spider methods (self.parse), so checkpoints work out of the box!
Other limitations:
  • Cookie expiration: If you wait too long to resume (days/weeks), cookies may expire and requests may fail. Resume within a reasonable timeframe (hours/days, not weeks).
  • Multiple runs: Each spider should have only one checkpoint at a time. Don’t run the same spider concurrently while a checkpoint exists.
  • Proxy type changes: If you change --proxy-type when resuming, the checkpoint is automatically cleared (see below).

​
Proxy Type Changes (Expert-in-the-Loop)

If you change --proxy-type when resuming, the checkpoint is automatically cleared and crawl starts fresh.
Example scenario:
1

Start crawl with auto mode

./scrapai crawl myspider --project proj
Uses datacenter proxies (auto mode default)
2

Datacenter fails, pause crawl

⚠️  EXPERT-IN-THE-LOOP: Datacenter proxy failed
🏠 To use residential proxy, run:
  ./scrapai crawl myspider --project proj --proxy-type residential
3

Resume with residential proxy

./scrapai crawl myspider --project proj --proxy-type residential
Output:
⚠️  Proxy type changed: auto → residential
🗑️  Clearing checkpoint to ensure all URLs retried with residential proxy
♻️  Starting fresh crawl
Why checkpoint is cleared:
  • Ensures blocked URLs are retried with new proxy type
  • Prevents Scrapy’s dupefilter from skipping already-seen failed URLs

​
When Checkpoints Are Useful

✅ Long-running crawls (hours/days) Resume if interrupted✅ Unstable connections Resume after network failures✅ System maintenance Pause before server restart, resume after✅ Resource management Pause during high-load periods, resume later

​
Technical Details

Built on Scrapy’s JOBDIR:
  • Uses Scrapy’s native pause/resume feature (not custom implementation)
  • Checkpoint files are pickle-serialized Scrapy objects
  • Atomic writes prevent checkpoint corruption
  • Compatible with all Scrapy spiders
Directory per spider:
  • Each spider gets its own checkpoint directory
  • Prevents conflicts between spiders
  • Clean separation of state
Smart cleanup:
  • Exit code 0 (success) → cleanup checkpoint
  • Exit code != 0 (error/Ctrl+C) → keep checkpoint for resume

​
Troubleshooting

​
Checkpoint Not Resuming

1

Check if checkpoint exists

ls -la ./data/myproject/myspider/checkpoint/
If directory doesn’t exist:
  • Checkpoint was cleaned up (successful completion)
  • Or never created (test mode with --limit)
2

Check for proxy type change

Changing --proxy-type clears checkpoint automatically

​
Start Fresh (Discard Checkpoint)

# Delete checkpoint directory
rm -rf ./data/myproject/myspider/checkpoint/

# Run crawl again
./scrapai crawl myspider --project myproject

​
Checkpoint from Old Spider Version

If you updated spider rules/selectors significantly, old checkpoint may be incompatible.
Solution: 
# Delete checkpoint and start fresh
rm -rf ./data/myproject/myspider/checkpoint/
./scrapai crawl myspider --project myproject

​
Checkpoint Files Too Large

Check size: 
du -sh ./data/myproject/myspider/checkpoint/
Large checkpoints indicate:
  • Many pending URLs (normal for large crawls)
  • Consider crawling in smaller batches
  • Or use incremental crawling (DeltaFetch)

Incremental Crawling

Skip unchanged pages on subsequent crawls

Queue Processing

Batch process multiple websites

Proxy Escalation

Smart proxy usage with cost control