Playwright Extractor

The Playwright extractor re-fetches each page in a cold headless browser, then runs trafilatura on the rendered HTML. It is a legacy strategy.

Legacy — not in the default extractor order. The default generic order is ["trafilatura", "newspaper"]. Playwright is only used when you list "playwright" explicitly in EXTRACTOR_ORDER, or when INFINITE_SCROLL is enabled (which re-adds it automatically).For JavaScript-heavy or anti-bot sites, the recommended path is the hybrid browser service — set "BROWSER_ENABLED": true (or "CLOUDFLARE_ENABLED": true for Cloudflare-protected sites). It renders and caches cookies instead of re-fetching every URL in a cold browser, so it is far faster at scale. See Cloudflare Bypass.Reach for Playwright directly only for infinite-scroll pages, or an existing spider already built on it.

When to Use

Infinite scroll pages (auto-enables Playwright via INFINITE_SCROLL)

An existing spider already pinned to "playwright" in EXTRACTOR_ORDER

For new JS-rendered or anti-bot sites, prefer BROWSER_ENABLED / CLOUDFLARE_ENABLED instead — see the warning above.

Slower than HTML-based extractors (newspaper, trafilatura) — a second request per URL

Higher resource usage (browser instance per request)

Use low concurrency (2-4 requests)

Configuration

Basic Playwright

{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 2
  }
}

Wait for Selector

{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".article-content",
    "CONCURRENT_REQUESTS": 2
  }
}

Waits for .article-content element (max 30s). Extraction fails if selector doesn’t appear.

Additional Delay

{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".loaded",
    "PLAYWRIGHT_DELAY": 5
  }
}

Waits for .loaded element, then waits an additional 5 seconds before extraction.

Infinite Scroll

{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".quote",
    "INFINITE_SCROLL": true,
    "MAX_SCROLLS": 10,
    "SCROLL_DELAY": 2.0
  },
  "rules": [
    {"allow": [".*"], "follow": false}
  ]
}

Settings Reference

PLAYWRIGHT_WAIT_SELECTOR

string

default:"null"

CSS selector to wait for before extractionTimeout: 30 secondsUse when: Content loads via AJAX/fetchExample:

"PLAYWRIGHT_WAIT_SELECTOR": ".article-content"

PLAYWRIGHT_DELAY

float

default:"null"

Additional seconds to wait after page loadUse when: Content appears with unpredictable timingExample:

"PLAYWRIGHT_DELAY": 5

INFINITE_SCROLL

boolean

default:"false"

Enable infinite scroll behaviorNote: Setting this auto-adds "playwright" to EXTRACTOR_ORDER if not already presentExample:

"INFINITE_SCROLL": true

MAX_SCROLLS

integer

default:"5"

Maximum scroll iterationsExample:

"MAX_SCROLLS": 10

SCROLL_DELAY

float

default:"1.0"

Delay between scrolls (seconds)Example:

"SCROLL_DELAY": 2.0

Identifying JS-Rendered Sites

Signs page needs Playwright:

Minimal HTML: View page source shows almost empty body

<div id="app"></div>
<script src="bundle.js"></script>

Content in script tags: Data exists only as JavaScript objects
Loading placeholders: “Loading…” text or spinner elements
Generic extractors fail: Newspaper/trafilatura return no content

Test without browser:

./scrapai inspect https://example.com/article --project proj
./scrapai analyze data/proj/spider/analysis/page.html

If HTML is minimal/empty → the page needs browser rendering. Prefer "BROWSER_ENABLED": true (or "CLOUDFLARE_ENABLED": true); use Playwright only for infinite scroll or an existing playwright-based spider.

Common Wait Selectors

/* Article loaded */
.article-content
.post-body
#main-content

/* Generic "loaded" markers */
.loaded
[data-loaded="true"]

/* Specific content */
.quote
.product-info
#posts

/* List items */
ul.items li:nth-child(3)  /* Wait for 3rd item */

Performance Optimization

Reduce Concurrency

{
  "CONCURRENT_REQUESTS": 2,  // Low concurrency for browser requests
  "DOWNLOAD_DELAY": 2
}

Browser instances use significant memory; too many concurrent browsers cause system overload.

Use Playwright Only When Needed

// Good: Try fast extractors first
"EXTRACTOR_ORDER": ["newspaper", "playwright"]

// Bad: Always use slow browser
"EXTRACTOR_ORDER": ["playwright"]

Hybrid Strategy

{
  "settings": {
    "EXTRACTOR_ORDER": ["trafilatura", "playwright"],
    "CONCURRENT_REQUESTS": 8
  }
}

Most pages use fast trafilatura; only JS-heavy pages trigger Playwright, keeping average concurrency high.

Examples

SPA (React/Vue/Angular)

{
  "name": "spa_blog",
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": "article.post",
    "PLAYWRIGHT_DELAY": 2,
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 3
  }
}

Delayed Content

{
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "custom"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".content-loaded",
    "PLAYWRIGHT_DELAY": 5,
    "CUSTOM_SELECTORS": {
      "title": "h1.title",
      "content": "div.main-content"
    }
  }
}

Infinite Scroll (Quotes Example)

{
  "name": "quotes_scroll",
  "source_url": "https://quotes.toscrape.com/scroll",
  "allowed_domains": ["quotes.toscrape.com"],
  "start_urls": ["https://quotes.toscrape.com/scroll"],
  "settings": {
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".quote",
    "INFINITE_SCROLL": true,
    "MAX_SCROLLS": 10,
    "SCROLL_DELAY": 2.0,
    "CONCURRENT_REQUESTS": 1,
    "DOWNLOAD_DELAY": 0
  },
  "rules": [
    {
      "allow": [".*"],
      "callback": "parse_article",
      "follow": false
    }
  ]
}

Note: Set follow: false to prevent following links during scroll.

Cloudflare + Playwright

{
  "settings": {
    "CLOUDFLARE_ENABLED": true,
    "CLOUDFLARE_STRATEGY": "browser_only",
    "EXTRACTOR_ORDER": ["playwright", "trafilatura"],
    "PLAYWRIGHT_WAIT_SELECTOR": ".article",
    "CONCURRENT_REQUESTS": 2,
    "DOWNLOAD_DELAY": 5
  }
}

Debugging

Check Playwright Logs

./scrapai crawl spider --limit 1 --project proj

Look for:

Starting Playwright fetch for https://...
Will wait for selector: .article-content
Will wait additional 2 seconds
Browser navigated
Got HTML from browser: 45678 bytes

Test Wait Selector

# Inspect page
./scrapai inspect https://example.com/spa-page --project proj

# Check if selector exists
./scrapai analyze page.html --test ".article-content"

If selector not found → Playwright will timeout (30s)

Common Issues

Timeout waiting for selector:

Selector doesn’t exist on page
Selector appears after 30s (use PLAYWRIGHT_DELAY instead)
JavaScript error prevents rendering
Solution: Use generic selector or remove PLAYWRIGHT_WAIT_SELECTOR

Content still empty:

Increase PLAYWRIGHT_DELAY
Page requires interaction - use INFINITE_SCROLL if applicable

Extractors Overview - Strategy selection
Trafilatura Extractor - Recommended after Playwright
Custom Extractors - For structured JS content
Settings - All Playwright settings

​When to Use

​Configuration

​Basic Playwright

​Wait for Selector

​Additional Delay

​Infinite Scroll

​Settings Reference

​Identifying JS-Rendered Sites

​Common Wait Selectors

​Performance Optimization

​Reduce Concurrency

​Use Playwright Only When Needed

​Hybrid Strategy

​Examples

​SPA (React/Vue/Angular)

​Delayed Content

​Infinite Scroll (Quotes Example)

​Cloudflare + Playwright

​Debugging

​Check Playwright Logs

​Test Wait Selector

​Common Issues

​Related

When to Use

Configuration

Basic Playwright

Wait for Selector

Additional Delay

Infinite Scroll

Settings Reference

Identifying JS-Rendered Sites

Common Wait Selectors

Performance Optimization

Reduce Concurrency

Use Playwright Only When Needed

Hybrid Strategy

Examples

SPA (React/Vue/Angular)

Delayed Content

Infinite Scroll (Quotes Example)

Cloudflare + Playwright

Debugging

Check Playwright Logs

Test Wait Selector

Common Issues

Related