> ## Documentation Index > Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt > Use this file to discover all available pages before exploring further. # Checkpoint Pause/Resume > Pause long-running crawls and resume them later without losing progress scrapai automatically enables checkpoint support for production crawls, allowing you to pause long-running crawls and resume them later without losing progress. ## How It Works Checkpoint is **automatically enabled** when running production crawls (no `--limit` flag) Checkpoint saved automatically when interrupted Automatically detects checkpoint and resumes from where you left off Checkpoint deleted automatically on successful completion **Test crawls** (with `--limit`) do **not** use checkpoints since they're short-running. ## What Gets Saved Scrapy's JOBDIR feature saves: 1. **Pending requests** - All URLs waiting to be crawled 2. **Duplicates filter** - URLs already visited (prevents re-crawling) 3. **Spider state** - Any custom state stored in `spider.state` dict ## Usage ### Production Crawl with Checkpoint ```bash theme={null} # Start production crawl (checkpoint auto-enabled) ./scrapai crawl myspider --project myproject ``` **Console output:** ```log theme={null} ๐Ÿ’พ Checkpoint enabled: ./data/myproject/myspider/checkpoint ``` **Pause the crawl:** ```bash theme={null} # Press Ctrl+C ^C ``` **Resume later:** ```bash theme={null} # Run same command ./scrapai crawl myspider --project myproject # Automatically detects checkpoint and resumes ``` ### Test Crawl (No Checkpoint) ```bash theme={null} # Test mode - no checkpoint needed (short run) ./scrapai crawl myspider --project myproject --limit 10 ``` **Console output:** ```log theme={null} ๐Ÿงช Test mode: Saving to database (limit: 10 items) ``` ## Checkpoint Storage Checkpoints are stored in your DATA\_DIR: ``` DATA_DIR///checkpoint/ ``` **Example directory structure:** ``` ./data/myproject/myspider/ โ”œโ”€โ”€ analysis/ # Phase 1-3 files โ”œโ”€โ”€ crawls/ # Production outputs โ”œโ”€โ”€ exports/ # Database exports โ””โ”€โ”€ checkpoint/ # Checkpoint state (auto-cleaned on success) ``` ## Cleanup **Manual cleanup:** ```bash theme={null} # If you want to discard a checkpoint and start fresh rm -rf ./data/myproject/myspider/checkpoint/ ``` ## Complete Example ```bash theme={null} ./scrapai crawl techcrunch --project news ``` **Output:** ```log theme={null} ๐Ÿ’พ Checkpoint enabled: ./data/news/techcrunch/checkpoint Crawling: https://techcrunch.com/ Scraped 50 items... Scraped 100 items... ``` ```bash theme={null} ^C ``` **Output:** ```log theme={null} Received interrupt signal, shutting down... Checkpoint saved: 150 items scraped, 237 URLs pending Run same command to resume from checkpoint ``` ```bash theme={null} ls -la ./data/news/techcrunch/checkpoint/ ``` **Output:** ``` drwxr-xr-x 5 user staff 160 Feb 24 10:30 . drwxr-xr-x 7 user staff 224 Feb 24 10:15 .. -rw-r--r-- 1 user staff 4096 Feb 24 10:30 requests.queue -rw-r--r-- 1 user staff 8192 Feb 24 10:30 dupefilter.db -rw-r--r-- 1 user staff 512 Feb 24 10:30 spider.state ``` ```bash theme={null} ./scrapai crawl techcrunch --project news ``` **Output:** ```log theme={null} โ™ป๏ธ Resuming from checkpoint: 150 items already scraped, 237 URLs pending Continuing crawl... Scraped 160 items... Scraped 200 items... Crawl completed successfully! ๐Ÿงน Checkpoint cleaned up (crawl completed) ``` ## Limitations **Request callbacks must be spider methods** (Scrapy limitation): ```python theme={null} # โœ… Works (spider method) Request(url, callback=self.parse_article) # โŒ Won't work (external function) Request(url, callback=some_external_function) ``` โœ… **scrapai spiders already compatible**: Our database spiders use spider methods (`self.parse`), so checkpoints work out of the box! Other limitations: * **Cookie expiration**: If you wait too long to resume (days/weeks), cookies may expire and requests may fail. Resume within a reasonable timeframe (hours/days, not weeks). * **Multiple runs**: Each spider should have only one checkpoint at a time. Don't run the same spider concurrently while a checkpoint exists. * **Proxy type changes**: If you change `--proxy-type` when resuming, the checkpoint is automatically cleared (see below). ## Proxy Type Changes (Expert-in-the-Loop) If you change `--proxy-type` when resuming, the checkpoint is automatically cleared and crawl starts fresh. **Example scenario:** ```bash theme={null} ./scrapai crawl myspider --project proj ``` Uses datacenter proxies (auto mode default) ```log theme={null} โš ๏ธ EXPERT-IN-THE-LOOP: Datacenter proxy failed ๐Ÿ  To use residential proxy, run: ./scrapai crawl myspider --project proj --proxy-type residential ``` ```bash theme={null} ./scrapai crawl myspider --project proj --proxy-type residential ``` **Output:** ```log theme={null} โš ๏ธ Proxy type changed: auto โ†’ residential ๐Ÿ—‘๏ธ Clearing checkpoint to ensure all URLs retried with residential proxy โ™ป๏ธ Starting fresh crawl ``` **Why checkpoint is cleared:** * Ensures blocked URLs are retried with new proxy type * Prevents Scrapy's dupefilter from skipping already-seen failed URLs ## When Checkpoints Are Useful โœ… **Long-running crawls** (hours/days) Resume if interrupted โœ… **Unstable connections** Resume after network failures โœ… **System maintenance** Pause before server restart, resume after โœ… **Resource management** Pause during high-load periods, resume later โŒ **Short test crawls** (minutes) Not needed, checkpoints disabled โŒ **Quick prototyping** Use `--limit` flag, no checkpoints ## Technical Details **Built on Scrapy's JOBDIR:** * Uses Scrapy's native pause/resume feature (not custom implementation) * Checkpoint files are pickle-serialized Scrapy objects * Atomic writes prevent checkpoint corruption * Compatible with all Scrapy spiders **Directory per spider:** * Each spider gets its own checkpoint directory * Prevents conflicts between spiders * Clean separation of state **Smart cleanup:** * Exit code 0 (success) โ†’ cleanup checkpoint * Exit code != 0 (error/Ctrl+C) โ†’ keep checkpoint for resume ## Troubleshooting ### Checkpoint Not Resuming ```bash theme={null} ls -la ./data/myproject/myspider/checkpoint/ ``` If directory doesn't exist: * Checkpoint was cleaned up (successful completion) * Or never created (test mode with `--limit`) Changing `--proxy-type` clears checkpoint automatically ### Start Fresh (Discard Checkpoint) ```bash theme={null} # Delete checkpoint directory rm -rf ./data/myproject/myspider/checkpoint/ # Run crawl again ./scrapai crawl myspider --project myproject ``` ### Checkpoint from Old Spider Version If you updated spider rules/selectors significantly, old checkpoint may be incompatible. **Solution:** ```bash theme={null} # Delete checkpoint and start fresh rm -rf ./data/myproject/myspider/checkpoint/ ./scrapai crawl myspider --project myproject ``` ### Checkpoint Files Too Large **Check size:** ```bash theme={null} du -sh ./data/myproject/myspider/checkpoint/ ``` **Large checkpoints indicate:** * Many pending URLs (normal for large crawls) * Consider crawling in smaller batches * Or use incremental crawling (DeltaFetch) ## Related Guides Skip unchanged pages on subsequent crawls Batch process multiple websites Smart proxy usage with cost control