> ## Documentation Index > Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt > Use this file to discover all available pages before exploring further. # Queue Management > Batch processing with database-backed queue and atomic locking The queue system enables batch processing of multiple websites with status tracking, priority ordering, and atomic claim operations. Perfect for processing hundreds of URLs with AI agents or parallel workers. ## queue add Add a single website to the queue. ### Syntax ```bash theme={null} ./scrapai queue add --project [options] ``` ### Arguments Website URL to add to queue. ### Options Project name. Custom instruction for processing this URL (e.g., "Focus on product pricing"). Priority level (higher = processed sooner). Range: 1-10. ### Examples ```bash theme={null} # Simple add ./scrapai queue add https://example.com --project myproject # With custom instructions ./scrapai queue add https://techsite.com --project tech \ --message "Extract author and publication date" # High priority ./scrapai queue add https://urgent.com --project news --priority 10 ``` ### Output ```bash theme={null} $ ./scrapai queue add https://bbc.co.uk --project news --priority 8 ✅ Added to queue (ID: 42) URL: https://bbc.co.uk Project: news Priority: 8 ``` ### Duplicate Handling Duplicates are detected by `(project_name, website_url)` combination and automatically skipped. ## queue bulk Bulk add URLs from CSV or JSON file. ### Syntax ```bash theme={null} ./scrapai queue bulk --project [options] ``` ### Arguments Path to CSV or JSON file containing URLs. ### Options Project name for all URLs. Default priority for all URLs (can be overridden per-row). ### CSV Format Requires a `url` column. Optional: `custom_instruction`, `priority`. ```csv theme={null} url,custom_instruction,priority https://bbc.co.uk,Focus on UK news,8 https://cnn.com,Focus on breaking news,9 https://reuters.com,,7 ``` ### JSON Format Array of objects with `url` field. Optional: `custom_instruction`, `priority`. ```json theme={null} [ { "url": "https://bbc.co.uk", "custom_instruction": "Focus on UK news", "priority": 8 }, { "url": "https://cnn.com", "custom_instruction": "Focus on breaking news", "priority": 9 }, { "url": "https://reuters.com", "priority": 7 } ] ``` ### Examples ```bash theme={null} # Bulk add from CSV ./scrapai queue bulk websites.csv --project news # Bulk add from JSON ./scrapai queue bulk urls.json --project ecommerce --priority 6 ``` ### Output ```bash theme={null} $ ./scrapai queue bulk news_sites.csv --project news ✅ Bulk add complete: Added: 47 Skipped (duplicates/invalid): 3 Project: news Format: CSV ``` Use `templates/queue-template.csv` as a starting point for your CSV files. ## queue list List queue items with filtering. ### Syntax ```bash theme={null} ./scrapai queue list --project [options] ``` ### Options Project name. Filter by status: `pending`, `processing`, `completed`, `failed`. Maximum items to show. Show all items including completed and failed (default shows only pending/processing). Show only the count, no details. ### Examples ```bash theme={null} # Show pending/processing items (default) ./scrapai queue list --project news # Show all items ./scrapai queue list --project news --all # Show only failed items ./scrapai queue list --project news --status failed # Count pending items ./scrapai queue list --project news --status pending --count ``` ### Output ```bash theme={null} $ ./scrapai queue list --project news 📋 Queue for project 'news': ⏳ [42] https://bbc.co.uk Status: pending | Priority: 8 Instructions: Focus on UK news 🔄 [43] https://cnn.com Status: processing | Priority: 9 Processing by: user@hostname (since 2026-02-28 15:30) ⏳ [44] https://reuters.com Status: pending | Priority: 7 ``` ## queue next Atomically claim the next pending item. ### Syntax ```bash theme={null} ./scrapai queue next --project ``` ### Options Project name. ### Behavior Atomically claims the highest priority pending item and locks it for processing. PostgreSQL uses `FOR UPDATE SKIP LOCKED` (race-safe). SQLite uses conditional UPDATE. ### Example ```bash theme={null} ./scrapai queue next --project news ``` ### Output #### Item Claimed ```bash theme={null} 🔄 Claimed item from queue: ID: 42 URL: https://bbc.co.uk Instructions: Focus on UK news Priority: 8 Locked by: user@hostname ``` #### Empty Queue ```bash theme={null} 📬 No pending items in queue for project 'news' ``` ## queue complete Mark an item as successfully completed. ### Syntax ```bash theme={null} ./scrapai queue complete ``` ### Arguments Queue item ID. ### Example ```bash theme={null} ./scrapai queue complete 42 ``` ### Output ```bash theme={null} ✅ Item 42 marked as completed URL: https://bbc.co.uk ``` ## queue fail Mark an item as failed with error message. ### Syntax ```bash theme={null} ./scrapai queue fail [--message ] ``` ### Arguments Queue item ID. ### Options Error message explaining why processing failed. ### Example ```bash theme={null} ./scrapai queue fail 42 --message "Site requires login" ``` ### Output ```bash theme={null} ❌ Item 42 marked as failed URL: https://bbc.co.uk Error: Site requires login ``` ## queue retry Reset a failed item to pending status. ### Syntax ```bash theme={null} ./scrapai queue retry ``` ### Arguments Queue item ID. ### Example ```bash theme={null} ./scrapai queue retry 42 ``` ### Output ```bash theme={null} 🔄 Item 42 reset to pending (retry count: 1) URL: https://bbc.co.uk ``` Retry count is tracked but not enforced. You can retry items indefinitely. ## queue remove Remove an item from the queue. ### Syntax ```bash theme={null} ./scrapai queue remove ``` ### Arguments Queue item ID. ### Example ```bash theme={null} ./scrapai queue remove 42 ``` ### Output ```bash theme={null} 🗑️ Item 42 removed from queue URL: https://bbc.co.uk ``` ## queue cleanup Bulk remove completed or failed items. ### Syntax ```bash theme={null} ./scrapai queue cleanup --project [options] ``` ### Options Project name. Remove all completed items. Remove all failed items. Remove all completed and failed items. Skip confirmation prompt. ### Examples ```bash theme={null} # Clean up completed items ./scrapai queue cleanup --project news --completed # Clean up failed items ./scrapai queue cleanup --project news --failed # Clean up everything (completed + failed) ./scrapai queue cleanup --project news --all --force ``` ### Output ```bash theme={null} $ ./scrapai queue cleanup --project news --all 🗑️ Found 47 items to remove: ✅ [42] https://bbc.co.uk ✅ [43] https://cnn.com ❌ [44] https://broken-site.com ✅ [45] https://reuters.com ... and 43 more Remove 47 items? (y/N): y ✅ Removed 47 items from queue ``` ## Status Values * `pending`: Waiting to be processed * `processing`: Currently being worked on (locked) * `completed`: Successfully processed * `failed`: Processing failed with error ## Parallel Processing Process queue items with multiple workers: ```bash theme={null} while true; do ./scrapai queue next --project news | grep "ID:" || break # Process the URL and mark complete/failed done ``` Use PostgreSQL for parallel processing. SQLite may have race conditions under high concurrency. ## Next Steps Process queue items with crawl commands Query and manage queue data