> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Queue Processing

> Batch process multiple websites with priority management and status tracking

Optional queue system for managing multiple website crawls. Use when explicitly needed. Always specify `--project` when working with queues.

## Commands Reference

### Add Single URL

```bash theme={null}
./scrapai queue add <url> --project NAME [-m "instruction"] [--priority N]
```

**Example:**

```bash theme={null}
./scrapai queue add https://example.com --project myproject -m "Focus on research articles" --priority 5
```

### Bulk Add URLs

```bash theme={null}
./scrapai queue bulk <file.csv|file.json> --project NAME [--priority N]
```

**Example:**

```bash theme={null}
./scrapai queue bulk sites.json --project myproject
```

### List Queue Items

<Tabs>
  <Tab title="Default (5 pending)">
    ```bash theme={null}
    ./scrapai queue list --project NAME
    ```

    Shows 5 pending/processing items by default
  </Tab>

  <Tab title="More Items">
    ```bash theme={null}
    ./scrapai queue list --project NAME --limit 20
    ```

    Show more queue items
  </Tab>

  <Tab title="All Statuses">
    ```bash theme={null}
    ./scrapai queue list --project NAME --all --limit 50
    ```

    Include completed and failed items
  </Tab>

  <Tab title="Filter by Status">
    ```bash theme={null}
    ./scrapai queue list --project NAME --status pending
    ./scrapai queue list --project NAME --status processing
    ./scrapai queue list --project NAME --status failed
    ```

    Show specific status only
  </Tab>

  <Tab title="Count Only">
    ```bash theme={null}
    ./scrapai queue list --project NAME --count
    ./scrapai queue list --project NAME --status failed --count
    ```

    Just show the count
  </Tab>
</Tabs>

### Claim Next Item

```bash theme={null}
./scrapai queue next --project NAME
```

Claims next highest-priority pending item and marks it as processing.

### Update Item Status

<Note>
  ID is globally unique, no `--project` needed for status updates
</Note>

<CodeGroup>
  ```bash Complete theme={null}
  ./scrapai queue complete <id>
  ```

  ```bash Fail theme={null}
  ./scrapai queue fail <id> [-m "error message"]
  ```

  ```bash Retry theme={null}
  ./scrapai queue retry <id>
  ```

  ```bash Remove theme={null}
  ./scrapai queue remove <id>
  ```
</CodeGroup>

### Cleanup Queue

<Warning>
  Cleanup commands require `--force` flag to prevent accidental deletion
</Warning>

<CodeGroup>
  ```bash Completed Items theme={null}
  ./scrapai queue cleanup --completed --force --project NAME
  ```

  ```bash Failed Items theme={null}
  ./scrapai queue cleanup --failed --force --project NAME
  ```

  ```bash All Items theme={null}
  ./scrapai queue cleanup --all --force --project NAME
  ```
</CodeGroup>

## Bulk File Formats

### JSON Format

```json queue.json theme={null}
[
  {
    "url": "https://site1.com",
    "custom_instruction": "Focus on research articles",
    "priority": 5
  },
  {
    "url": "https://site2.com",
    "priority": 10
  },
  {
    "url": "https://site3.com",
    "custom_instruction": "Include all news sections",
    "priority": 1
  }
]
```

<Note>
  Template available: `templates/queue-template.json`
</Note>

### CSV Format

```csv queue.csv theme={null}
url,custom_instruction,priority
https://site1.com,Focus on research articles,5
https://site2.com,,10
https://site3.com,Include all news sections,1
```

<Note>
  Template available: `templates/queue-template.csv`
</Note>

**Required columns:**

* `url` - Website URL (required)
* `custom_instruction` - Special instructions for this site (optional)
* `priority` - Higher number = higher priority (optional, default: 0)

## Queue Processing Workflow

<Steps>
  <Step title="Claim next item">
    ```bash theme={null}
    ./scrapai queue next --project NAME
    ```

    Note the returned values:

    * ID (for status updates)
    * URL (target website)
    * Project name
    * Custom instruction (if exists)
  </Step>

  <Step title="Process custom instruction">
    If `custom_instruction` exists, use it to override default analysis behavior (e.g., focus on specific content types, custom selectors).
  </Step>

  <Step title="Run analysis phases">
    Execute standard analysis workflow (Phases 1-4).

    Include `"source_url": "<queue_url>"` in `final_spider.json`.
  </Step>

  <Step title="Import spider">
    ```bash theme={null}
    ./scrapai import final_spider.json --project myproject
    ```

    <Warning>
      Always use `--project` matching the queue item's project! Without it, spider defaults to "default" project, mixing data across projects.
    </Warning>
  </Step>

  <Step title="Run crawl">
    ```bash theme={null}
    ./scrapai crawl spider_name --project myproject
    ```
  </Step>

  <Step title="Update status on success">
    ```bash theme={null}
    ./scrapai queue complete <id>
    ```
  </Step>

  <Step title="Update status on failure">
    ```bash theme={null}
    ./scrapai queue fail <id> -m "error description"
    ```
  </Step>
</Steps>

## Complete Example

### 1. Create Queue File

```json sites.json theme={null}
[
  {
    "url": "https://techcrunch.com",
    "custom_instruction": "Focus on AI and startup news",
    "priority": 10
  },
  {
    "url": "https://theverge.com",
    "custom_instruction": "All technology articles",
    "priority": 5
  },
  {
    "url": "https://arstechnica.com",
    "priority": 3
  }
]
```

### 2. Bulk Add to Queue

```bash theme={null}
./scrapai queue bulk sites.json --project tech_news
```

### 3. Check Queue

```bash theme={null}
./scrapai queue list --project tech_news
```

**Output:**

```
Queue items for project 'tech_news':

ID: 1
URL: https://techcrunch.com
Status: pending
Priority: 10
Custom instruction: Focus on AI and startup news
Created: 2024-02-24 10:30:00

... (2 more items)
```

### 4. Process Queue

```bash theme={null}
# Claim next (highest priority)
./scrapai queue next --project tech_news

# Output:
# Claimed queue item #1
# Project: tech_news
# URL: https://techcrunch.com
# Custom instruction: Focus on AI and startup news
# Priority: 10

# Run analysis workflow (Phases 1-4)
./scrapai inspect https://techcrunch.com --project tech_news
# ... complete analysis phases ...

# Import spider
./scrapai import final_spider.json --project tech_news

# Test crawl
./scrapai crawl techcrunch --project tech_news --limit 10

# Mark as complete
./scrapai queue complete 1
```

### 5. Handle Failure

```bash theme={null}
# If crawl fails
./scrapai queue fail 1 -m "Site requires authentication"

# Check failed items
./scrapai queue list --project tech_news --status failed

# Retry if needed
./scrapai queue retry 1
```

### 6. Cleanup

```bash theme={null}
# Remove completed items
./scrapai queue cleanup --completed --force --project tech_news

# Or remove all
./scrapai queue cleanup --all --force --project tech_news
```

## Statistics

Check queue statistics:

```bash theme={null}
# Count by status
./scrapai queue list --project tech_news --status pending --count
./scrapai queue list --project tech_news --status completed --count
./scrapai queue list --project tech_news --status failed --count

# View all items
./scrapai queue list --project tech_news --all --limit 100
```

## Best Practices

<Steps>
  <Step title="Use descriptive custom instructions">
    Help guide the analysis process with clear instructions:

    ```bash theme={null}
    ./scrapai queue add https://example.com --project proj -m "Focus on research papers, ignore blog posts"
    ```
  </Step>

  <Step title="Set meaningful priorities">
    Higher numbers processed first:

    * 10: Urgent, high-value sites
    * 5: Standard priority
    * 1: Low priority, process when available
  </Step>

  <Step title="Always specify project">
    Prevent data mixing across projects:

    ```bash theme={null}
    ./scrapai import spider.json --project myproject
    ./scrapai crawl spider --project myproject
    ```
  </Step>

  <Step title="Document failures">
    Help debug issues later:

    ```bash theme={null}
    ./scrapai queue fail <id> -m "Requires JavaScript rendering - use --browser flag"
    ```
  </Step>

  <Step title="Regular cleanup">
    Remove old completed items:

    ```bash theme={null}
    ./scrapai queue cleanup --completed --force --project myproject
    ```
  </Step>
</Steps>

## Related Guides

<CardGroup cols={2}>
  <Card title="Checkpoint Resume" icon="save" href="/guides/checkpoint-resume">
    Pause and resume long crawls
  </Card>

  <Card title="Incremental Crawling" icon="rotate" href="/guides/incremental-crawling">
    Skip unchanged pages
  </Card>
</CardGroup>