> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Content Extractors

> Configure extraction strategies for different content types and rendering methods

Test generic extractors (newspaper, trafilatura) first. Only use custom selectors if they fail.

## Discovery Workflow

<Steps>
  <Step title="Inspect article page">
    ```bash theme={null}
    ./scrapai inspect https://example.com/article-url --project proj
    ```
  </Step>

  <Step title="Analyze HTML structure">
    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html
    ```
  </Step>

  <Step title="Test selectors">
    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html --test "h1.article-title"
    ./scrapai analyze data/proj/spider/analysis/page.html --test "div.article-body"
    ```
  </Step>

  <Step title="Search for specific fields">
    ```bash theme={null}
    ./scrapai analyze data/proj/spider/analysis/page.html --find "price"
    ```
  </Step>
</Steps>

## Extractor Order Options

| Config                                   | When to use                                          |
| ---------------------------------------- | ---------------------------------------------------- |
| `["newspaper", "trafilatura"]`           | Generic extractors work (clean news/blog HTML)       |
| `["custom", "newspaper", "trafilatura"]` | Generic extractors fail; custom selectors needed     |
| `["playwright", "custom"]`               | JS-rendered content (SPAs, dynamic loading)          |
| `["playwright", "trafilatura"]`          | JS-rendered, generic extractors work after rendering |

## Generic Extractors

<Tabs>
  <Tab title="Newspaper">
    **Best for:** Clean news articles and blog posts

    ```json theme={null}
    {
      "settings": {
        "EXTRACTOR_ORDER": ["newspaper", "trafilatura"]
      }
    }
    ```

    **What it extracts:**

    * Title
    * Author
    * Published date
    * Main content
    * Top image
    * Keywords
    * Summary
  </Tab>

  <Tab title="Trafilatura">
    **Best for:** Wide variety of content types

    ```json theme={null}
    {
      "settings": {
        "EXTRACTOR_ORDER": ["trafilatura", "newspaper"]
      }
    }
    ```

    **What it extracts:**

    * Title
    * Author
    * Published date
    * Main content
    * Description
    * Site name
    * Categories
    * Tags
  </Tab>
</Tabs>

<Note>
  Generic extractors work for \~80% of news/blog sites. Try them first before creating custom selectors.
</Note>

## Custom Selectors

**Standard fields** (title, author, content, date) → main DB columns.
**Any other field** → stored in `metadata` JSON column.

### News Article

```json spider.json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["custom", "trafilatura"],
    "CUSTOM_SELECTORS": {
      "title": "h1.article-title",
      "content": "div.article-body",
      "author": "span.author-name",
      "date": "time.published-date",
      "category": "a.category-link",
      "tags": "div.tags a"
    }
  }
}
```

### E-commerce Product

```json spider.json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["custom"],
    "CUSTOM_SELECTORS": {
      "title": "h1.product-name",
      "content": "div.product-description",
      "price": "span.price-value",
      "rating": "div.star-rating",
      "stock": "span.availability",
      "brand": "div.brand-name"
    }
  }
}
```

### Forum Thread

```json spider.json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["custom"],
    "CUSTOM_SELECTORS": {
      "title": "h1.thread-title",
      "author": "span.username",
      "content": "div.post-content",
      "date": "time.post-date",
      "upvotes": "span.vote-count"
    }
  }
}
```

## Playwright Extractor

### Basic Configuration

```json spider.json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".article-content",
    "PLAYWRIGHT_DELAY": 5
  }
}
```

**Settings:**

* `PLAYWRIGHT_WAIT_SELECTOR`: CSS selector to wait for (max 30s)
* `PLAYWRIGHT_DELAY`: Extra seconds after page load

### Infinite Scroll

```json spider.json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".quote",
    "PLAYWRIGHT_DELAY": 2,
    "INFINITE_SCROLL": true,
    "MAX_SCROLLS": 10,
    "SCROLL_DELAY": 2.0
  }
}
```

**Settings:**

* `INFINITE_SCROLL`: Enable scroll behavior (default: false)
* `MAX_SCROLLS`: Max scrolls to perform (default: 5)
* `SCROLL_DELAY`: Seconds between scrolls (default: 1.0)

### Complete Playwright Example

```json spa_spider.json theme={null}
{
  "name": "react_app",
  "allowed_domains": ["example.com"],
  "start_urls": ["https://example.com/app"],
  "rules": [
    {
      "allow": ["/article/.*"],
      "callback": "parse_article",
      "follow": false
    }
  ],
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "custom"],
    "PLAYWRIGHT_WAIT_SELECTOR": "#article-loaded",
    "PLAYWRIGHT_DELAY": 3,
    "CUSTOM_SELECTORS": {
      "title": "h1.title",
      "content": "div.content",
      "author": "span.author"
    }
  }
}
```

## Selector Discovery Principles

* Target main content element (not navigation, sidebar, footer)
* Selector should match ONE unique element
* Prefer specific classes (`.article-title` over `.title`)
* Test on multiple pages
* Prefer semantic tags (`<article>`, `<time>`, `<h1>`)
* Validate content length (>500 chars for content, >10 for title)
* Avoid dynamic/generated class names

<Warning>
  **Common mistakes:**

  * Selector matches multiple elements
  * Targets sidebar/footer instead of main content
  * Overly generic selectors like `div.text`
</Warning>

## Identifying JS-Rendered Sites

<CodeGroup>
  ```html Empty Content theme={null}
  <body>
    <div id="app"></div>
    <script src="bundle.js"></script>
  </body>
  ```

  ```html JS Data Objects theme={null}
  <script>
  window.__INITIAL_STATE__ = {"articles": [...]};
  </script>
  ```

  ```html Loading Placeholders theme={null}
  <div class="loading">Loading...</div>
  ```
</CodeGroup>

## Playwright Wait: Common Selectors

**Article sites:**

* `.article-content`
* `#main-content`
* `article.post`
* `[data-loaded="true"]`

**Product pages:**

* `.product-details`
* `.price-container`
* `#product-info`

**Social/Forums:**

* `.post-list`
* `#posts`
* `.loaded`

**Generic:**

* `.content-loaded`
* `[data-ready]`
* `.main-container`

## Implementation Details

**Extractor classes (from source: `core/extractors.py`):**

<CardGroup cols={3}>
  <Card title="NewspaperExtractor" icon="newspaper">
    `extractors.py:36-78`

    Uses newspaper4k library
  </Card>

  <Card title="TrafilaturaExtractor" icon="code">
    `extractors.py:80-128`

    Uses trafilatura library
  </Card>

  <Card title="CustomExtractor" icon="wand-magic-sparkles">
    `extractors.py:131-257`

    Uses BeautifulSoup + CSS selectors
  </Card>

  <Card title="SmartExtractor" icon="brain">
    `extractors.py:259-464`

    Tries multiple strategies in order
  </Card>

  <Card title="PlaywrightExtractor" icon="browser">
    `extractors.py:398-464`

    Async browser rendering
  </Card>
</CardGroup>

## Troubleshooting

### Generic Extractors Return Empty Content

1. Check if JS-rendered (empty `<div id="app"></div>`)
2. Try Playwright: `{"EXTRACTOR_ORDER": ["playwright", "trafilatura"]}`
3. Use custom selectors: `{"EXTRACTOR_ORDER": ["custom", "trafilatura"]}`

### Custom Selector Returns None

1. Test selector: `./scrapai analyze page.html --test "your-selector"`
2. Check selector specificity and uniqueness
3. Verify element exists in HTML structure

### Playwright Timeout

1. Increase delay: `{"PLAYWRIGHT_DELAY": 10}`
2. Use different wait selector that appears earlier
3. Verify selector exists in rendered page

### Content Extracted But Wrong

1. Verify selector uniqueness (may match sidebar/footer)
2. Make selector more specific: `{"content": "main article.post div.body"}`
3. Test on multiple pages

## Best Practices

1. Start with generic extractors: `["newspaper", "trafilatura"]` (80% success rate)
2. Add custom selectors if needed: `["custom", "trafilatura"]`
3. Use Playwright for JS-rendered sites: `["playwright", "trafilatura"]`
4. Test on multiple pages
5. Monitor quality: `./scrapai show 1 --project proj`

## Related Guides

<CardGroup cols={2}>
  <Card title="Custom Callbacks" icon="code" href="/guides/custom-callbacks">
    Extract structured data with callbacks
  </Card>

  <Card title="Data Processors" icon="wand-magic-sparkles" href="/guides/data-processors">
    Transform extracted data
  </Card>
</CardGroup>