> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Playwright Extractor

> Browser rendering for JavaScript-heavy sites

The Playwright extractor re-fetches each page in a cold headless browser, then runs trafilatura on the rendered HTML. It is a **legacy** strategy.

<Warning>
  **Legacy — not in the default extractor order.** The default generic order is `["trafilatura", "newspaper"]`. Playwright is only used when you list `"playwright"` explicitly in `EXTRACTOR_ORDER`, or when `INFINITE_SCROLL` is enabled (which re-adds it automatically).

  For JavaScript-heavy or anti-bot sites, the recommended path is the **hybrid browser service** — set `"BROWSER_ENABLED": true` (or `"CLOUDFLARE_ENABLED": true` for Cloudflare-protected sites). It renders and caches cookies instead of re-fetching every URL in a cold browser, so it is far faster at scale. See [Cloudflare Bypass](/guides/cloudflare-bypass).

  Reach for Playwright directly only for infinite-scroll pages, or an existing spider already built on it.
</Warning>

## When to Use

<Check>Infinite scroll pages (auto-enables Playwright via `INFINITE_SCROLL`)</Check>
<Check>An existing spider already pinned to `"playwright"` in `EXTRACTOR_ORDER`</Check>

<Note>For new JS-rendered or anti-bot sites, prefer `BROWSER_ENABLED` / `CLOUDFLARE_ENABLED` instead — see the warning above.</Note>

<Warning>Slower than HTML-based extractors (newspaper, trafilatura) — a second request per URL</Warning>
<Warning>Higher resource usage (browser instance per request)</Warning>
<Warning>Use low concurrency (2-4 requests)</Warning>

## Configuration

### Basic Playwright

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 2
  }
}
```

### Wait for Selector

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".article-content",
    "CONCURRENT_REQUESTS": 2
  }
}
```

Waits for `.article-content` element (max 30s). Extraction fails if selector doesn't appear.

### Additional Delay

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".loaded",
    "PLAYWRIGHT_DELAY": 5
  }
}
```

Waits for `.loaded` element, then waits an additional 5 seconds before extraction.

### Infinite Scroll

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".quote",
    "INFINITE_SCROLL": true,
    "MAX_SCROLLS": 10,
    "SCROLL_DELAY": 2.0
  },
  "rules": [
    {"allow": [".*"], "follow": false}
  ]
}
```

## Settings Reference

<ParamField path="PLAYWRIGHT_WAIT_SELECTOR" type="string" default="null">
  CSS selector to wait for before extraction

  **Timeout:** 30 seconds

  **Use when:** Content loads via AJAX/fetch

  **Example:**

  ```json theme={null}
  "PLAYWRIGHT_WAIT_SELECTOR": ".article-content"
  ```
</ParamField>

<ParamField path="PLAYWRIGHT_DELAY" type="float" default="null">
  Additional seconds to wait after page load

  **Use when:** Content appears with unpredictable timing

  **Example:**

  ```json theme={null}
  "PLAYWRIGHT_DELAY": 5
  ```
</ParamField>

<ParamField path="INFINITE_SCROLL" type="boolean" default="false">
  Enable infinite scroll behavior

  **Note:** Setting this auto-adds `"playwright"` to `EXTRACTOR_ORDER` if not already present

  **Example:**

  ```json theme={null}
  "INFINITE_SCROLL": true
  ```
</ParamField>

<ParamField path="MAX_SCROLLS" type="integer" default="5">
  Maximum scroll iterations

  **Example:**

  ```json theme={null}
  "MAX_SCROLLS": 10
  ```
</ParamField>

<ParamField path="SCROLL_DELAY" type="float" default="1.0">
  Delay between scrolls (seconds)

  **Example:**

  ```json theme={null}
  "SCROLL_DELAY": 2.0
  ```
</ParamField>

## Identifying JS-Rendered Sites

**Signs page needs Playwright:**

1. **Minimal HTML:** View page source shows almost empty body
   ```html theme={null}
   <div id="app"></div>
   <script src="bundle.js"></script>
   ```

2. **Content in script tags:** Data exists only as JavaScript objects

3. **Loading placeholders:** "Loading..." text or spinner elements

4. **Generic extractors fail:** Newspaper/trafilatura return no content

**Test without browser:**

```bash theme={null}
./scrapai inspect https://example.com/article --project proj
./scrapai analyze data/proj/spider/analysis/page.html
```

If HTML is minimal/empty → the page needs browser rendering. Prefer `"BROWSER_ENABLED": true` (or `"CLOUDFLARE_ENABLED": true`); use Playwright only for infinite scroll or an existing playwright-based spider.

## Common Wait Selectors

```css theme={null}
/* Article loaded */
.article-content
.post-body
#main-content

/* Generic "loaded" markers */
.loaded
[data-loaded="true"]

/* Specific content */
.quote
.product-info
#posts

/* List items */
ul.items li:nth-child(3)  /* Wait for 3rd item */
```

## Performance Optimization

### Reduce Concurrency

```json theme={null}
{
  "CONCURRENT_REQUESTS": 2,  // Low concurrency for browser requests
  "DOWNLOAD_DELAY": 2
}
```

Browser instances use significant memory; too many concurrent browsers cause system overload.

### Use Playwright Only When Needed

```json theme={null}
// Good: Try fast extractors first
"EXTRACTOR_ORDER": ["newspaper", "playwright"]

// Bad: Always use slow browser
"EXTRACTOR_ORDER": ["playwright"]
```

### Hybrid Strategy

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["trafilatura", "playwright"],
    "CONCURRENT_REQUESTS": 8
  }
}
```

Most pages use fast trafilatura; only JS-heavy pages trigger Playwright, keeping average concurrency high.

## Examples

### SPA (React/Vue/Angular)

```json theme={null}
{
  "name": "spa_blog",
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": "article.post",
    "PLAYWRIGHT_DELAY": 2,
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 3
  }
}
```

### Delayed Content

```json theme={null}
{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "custom"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".content-loaded",
    "PLAYWRIGHT_DELAY": 5,
    "CUSTOM_SELECTORS": {
      "title": "h1.title",
      "content": "div.main-content"
    }
  }
}
```

### Infinite Scroll (Quotes Example)

```json theme={null}
{
  "name": "quotes_scroll",
  "source_url": "https://quotes.toscrape.com/scroll",
  "allowed_domains": ["quotes.toscrape.com"],
  "start_urls": ["https://quotes.toscrape.com/scroll"],
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".quote",
    "INFINITE_SCROLL": true,
    "MAX_SCROLLS": 10,
    "SCROLL_DELAY": 2.0,
    "CONCURRENT_REQUESTS": 1,
    "DOWNLOAD_DELAY": 0
  },
  "rules": [
    {
      "allow": [".*"],
      "callback": "parse_article",
      "follow": false
    }
  ]
}
```

**Note:** Set `follow: false` to prevent following links during scroll.

### Cloudflare + Playwright

```json theme={null}
{
  "settings": {
    "CLOUDFLARE_ENABLED": true,
    "CLOUDFLARE_STRATEGY": "browser_only",
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".article",
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 5
  }
}
```

## Debugging

### Check Playwright Logs

```bash theme={null}
./scrapai crawl spider --limit 1 --project proj
```

Look for:

```
Starting Playwright fetch for https://...
Will wait for selector: .article-content
Will wait additional 2 seconds
Browser navigated
Got HTML from browser: 45678 bytes
```

### Test Wait Selector

```bash theme={null}
# Inspect page
./scrapai inspect https://example.com/spa-page --project proj

# Check if selector exists
./scrapai analyze page.html --test ".article-content"
```

If selector not found → Playwright will timeout (30s)

### Common Issues

**Timeout waiting for selector:**

* Selector doesn't exist on page
* Selector appears after 30s (use `PLAYWRIGHT_DELAY` instead)
* JavaScript error prevents rendering
* **Solution:** Use generic selector or remove `PLAYWRIGHT_WAIT_SELECTOR`

**Content still empty:**

* Increase `PLAYWRIGHT_DELAY`
* Page requires interaction - use `INFINITE_SCROLL` if applicable

## Related

* [Extractors Overview](/api/extractors-overview) - Strategy selection
* [Trafilatura Extractor](/api/trafilatura) - Recommended after Playwright
* [Custom Extractors](/api/custom-extractors) - For structured JS content
* [Settings](/api/settings) - All Playwright settings