> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# 4-Phase Workflow

> Complete workflow documentation for building scrapers with AI agents

Every spider goes through 4 phases, executed sequentially and completely. The agent never skips steps—each phase builds on the previous one.

<Note>
  This page summarizes the workflow. The authoritative, always-current version is [`CLAUDE.md`](https://github.com/discourselab/scrapai-cli/blob/main/CLAUDE.md) on `main` — the exact instructions the agent runs.
</Note>

<Steps>
  <Step title="Phase 1: Analysis & Section Documentation">
    Understand site structure, discover all content sections, document URL patterns
  </Step>

  <Step title="Phase 2: Rule Generation & Extraction Testing">
    Create URL matching rules, choose extraction strategy, test on sample pages
  </Step>

  <Step title="Phase 3: Prepare Spider Configuration">
    Create test and final spider JSON files with all rules and settings
  </Step>

  <Step title="Phase 4: Execution & Verification">
    Test extraction quality on sample articles, import final spider for production
  </Step>
</Steps>

**Only mark queue items complete when ALL phases pass.** If any fail: `./scrapai queue fail <id> -m "reason"`.

## Phase 1: Analysis & Section Documentation

**Goal:** Understand site structure, discover all content sections, document URL patterns.

### For Non-Sitemap URLs

```bash theme={null}
./scrapai inspect https://site.com/ --project proj
./scrapai extract-urls --file data/proj/spider/analysis/page.html --output data/proj/spider/analysis/all_urls.txt
```

Review `all_urls.txt`, categorize URLs (content/navigation/utility), drill into sections with `inspect` + `analyze`, document in `sections.md`.

### Exclusion Policy

**ONLY exclude:**

* About, contact, donate, account, legal, search pages
* PDFs and non-HTML files

**Everything else: explore and include.** When uncertain, include it.

**User instructions always override defaults.**

### For Sitemap URLs

If the URL points to an XML sitemap (e.g., `https://site.com/sitemap.xml`):

1. Inspect the sitemap to understand structure
2. Identify URL patterns for content pages
3. Use `USE_SITEMAP: true` in spider config
4. See [sitemap documentation](/essentials/sitemap) for details

### Phase 1 Complete When:

<Check>
  * `sections.md` exists in `data/<project>/<spider>/analysis/`
  * ALL content section types identified (blog, news, reports, etc.)
  * URL pattern documented for EACH section type
  * Example URLs listed (minimum 3 per section) for Phase 2 testing
  * Exclusions documented
</Check>

## Phase 2: Rule Generation & Extraction Testing

**Goal:** Create URL matching rules, choose extraction strategy (generic extractors, custom selectors, or callbacks).

### Decision Point: What Type of Content?

<CardGroup cols={2}>
  <Card title="Articles/Blog Posts" icon="newspaper">
    Use `parse_article` with generic extractors (newspaper, trafilatura)
  </Card>

  <Card title="Products/Jobs/Listings" icon="grid">
    Use named callbacks with custom fields
  </Card>
</CardGroup>

### For Article Content (title/content/author/date)

<Steps>
  <Step title="Create rules from sections.md">
    Use the URL patterns documented in Phase 1 to create rules for each section.
  </Step>

  <Step title="Test generic extractors">
    Inspect an article page and analyze its structure:

    ```bash theme={null}
    # Default: lightweight HTTP (works for most sites)
    ./scrapai inspect https://website.com/article-url --project proj

    # Use --browser if site needs JavaScript
    ./scrapai inspect https://website.com/article-url --project proj --browser

    # Use --browser if site is protected
    ./scrapai inspect https://website.com/article-url --project proj --browser

    ./scrapai analyze data/proj/spider/analysis/page.html
    ```

    If it has clean `<article>` tags / semantic HTML → generic extractors work.
  </Step>

  <Step title="If generic extractors fail">
    Discover custom CSS selectors using `./scrapai analyze`:

    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html
    ./scrapai analyze data/proj/spider/analysis/page.html --test "h1.article-title"
    ./scrapai analyze data/proj/spider/analysis/page.html --find "price"
    ```

    See [extractor documentation](/essentials/extractors) for selector discovery.
  </Step>

  <Step title="Consolidate into final_spider.json">
    Create the complete spider config with all rules and settings.
  </Step>
</Steps>

### For Non-Article Content (products, jobs, etc.)

<Steps>
  <Step title="Analyze a sample page">
    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html
    ```
  </Step>

  <Step title="Identify all fields to extract">
    For e-commerce: name, price, rating, availability, images

    For jobs: title, company, salary, location, description

    For real estate: address, price, bedrooms, square footage, features
  </Step>

  <Step title="Discover CSS selectors for each field">
    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html --test "h1.product-name::text"
    ./scrapai analyze data/proj/spider/analysis/page.html --find "price"
    ```
  </Step>

  <Step title="Create callback config">
    Build callback config with all fields + processors (CSS selectors, processors for cleaning/casting). See [callbacks documentation](/essentials/callbacks).
  </Step>

  <Step title="Test on multiple example pages">
    Verify selectors work across 2-3 different items to ensure consistency.
  </Step>

  <Step title="Consolidate into final_spider.json">
    Create the complete spider config with callbacks section.
  </Step>
</Steps>

### Phase 2 Complete When:

<Check>
  * `final_spider.json` created with all URL matching rules
  * Extractor strategy chosen:
    * **Generic extractors:** `EXTRACTOR_ORDER` configured
    * **Custom selectors:** `CUSTOM_SELECTORS` for title, content, author, date
    * **Named callbacks:** `callbacks` dict with custom field extraction
  * All settings documented (Cloudflare, Playwright, etc. if needed)
</Check>

## Phase 3: Prepare Spider Configuration

**Goal:** Create test and final spider JSON files with all rules and settings.

**Test spider config (`test_spider.json`):** 5 sample article URLs, `follow: false` on all rules (no crawling), same extractor settings as final.

**Final spider config (`final_spider.json`):** Full start\_urls, all URL matching rules with proper `follow` settings, complete extractor/callback configuration, all spider settings. Include `source_url` when processing from queue.

<Warning>
  **Do NOT import yet.** Importing happens in Phase 4 after validation.
</Warning>

### Phase 3 Complete When:

<Check>
  * `test_spider.json` created with 5 article URLs, `follow: false`
  * `final_spider.json` created with all start\_urls, rules, and settings
  * `source_url` included in config (if processing from queue)
</Check>

## Phase 4: Execution & Verification

**Goal:** Test extraction quality on sample articles, then import final spider for production.

### Step 4A: Test Extraction (5 Articles)

<Steps>
  <Step title="Import test spider">
    ```bash theme={null}
    ./scrapai spiders import test_spider.json --project proj
    ```
  </Step>

  <Step title="Run test crawl">
    ```bash theme={null}
    ./scrapai crawl spider_name --limit 5 --project proj
    ```

    Crawls exactly 5 URLs and saves results to database.
  </Step>

  <Step title="Verify output">
    ```bash theme={null}
    ./scrapai show spider_name --limit 5 --project proj
    ```

    Check that all fields are extracted correctly:

    * Title present and accurate
    * Content complete (not truncated)
    * Author extracted (if available)
    * Date parsed correctly
  </Step>

  <Step title="Fix if needed">
    If extraction is bad:

    * Review selectors in Phase 2
    * Update `test_spider.json`
    * Re-import and re-test

    **Only proceed when extraction is good.**
  </Step>
</Steps>

### Step 4B: Import Final Spider

<Steps>
  <Step title="Import final spider">
    ```bash theme={null}
    ./scrapai spiders import final_spider.json --project proj
    ```

    Using the same spider name auto-updates the existing config.
  </Step>

  <Step title="Spider is ready for production">
    The spider is now in the database and ready for full crawls.

    **Do NOT run production crawls yourself** as they can take hours or days.
  </Step>
</Steps>

### Production Crawls (User Runs)

<Warning>
  **Agent always uses `--limit 5` for testing. User runs production crawls without `--limit`.** Production crawls can take hours/days.
</Warning>

If user asks for full crawl: Explain it can take hours/days, provide command: `./scrapai crawl <spider_name> --project <project_name>`, mention checkpoint support (Ctrl+C to pause/resume).

### Phase 4 Complete When:

<Check>
  * Test crawl completed with `--limit 5`
  * `show` output verified: title, content, author, date extracted correctly
  * Final spider imported to database
  * Spider ready for production (user will run full crawl)
</Check>

## Settings Reference

* **Generic extractors:** `EXTRACTOR_ORDER: ["newspaper", "trafilatura"]` - clean semantic HTML
* **Custom selectors:** `CUSTOM_SELECTORS` with CSS selectors - when generic extractors fail
* **JavaScript-rendered:** `EXTRACTOR_ORDER: ["playwright", "custom"]` with wait selectors
* **Cloudflare:** `CLOUDFLARE_ENABLED: true` (test without `--browser` first) - see [Cloudflare docs](/essentials/cloudflare)
* **Sitemap:** `USE_SITEMAP: true` - see [sitemap docs](/essentials/sitemap)
* **DeltaFetch:** `DELTAFETCH_ENABLED: true` - skip already-scraped URLs (80-90% bandwidth reduction)
* **Infinite scroll:** `INFINITE_SCROLL: true, MAX_SCROLLS: 5`

## Parallel Queue Processing

Max 5 websites in parallel (e.g., 12 websites → batches of 5+5+2). Each website goes through Phase 1→2→3→4 sequentially, but multiple websites can be at different phases. Report progress per batch. See [queue documentation](/essentials/queue).

## Next Steps

<CardGroup cols={2}>
  <Card title="CLI Reference" icon="terminal" href="/cli/overview">
    Complete CLI command documentation
  </Card>

  <Card title="Extractors Guide" icon="code" href="/guides/extractors">
    Learn about extraction strategies and selector discovery
  </Card>

  <Card title="Custom Callbacks" icon="function" href="/guides/custom-callbacks">
    Custom field extraction for non-article content
  </Card>

  <Card title="Cloudflare Bypass" icon="shield" href="/guides/cloudflare-bypass">
    Bypass Cloudflare protection with cookie caching
  </Card>
</CardGroup>