> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloudflare Bypass

> Handle Cloudflare-protected sites with browser verification and cookie caching

Only enable Cloudflare bypass when the site explicitly requires it. Always test WITHOUT `--browser` first.

## Detection Indicators

Your site needs Cloudflare bypass if you see:

<Warning>
  * "Checking your browser" or "Just a moment" messages
  * 403/503 HTTP errors with Cloudflare branding
  * Challenge pages before content loads
</Warning>

<Warning>
  **VPS/Cloud Server IP Reputation Issue:**

  If running on AWS, DigitalOcean, Hetzner, or any cloud provider, Cloudflare may block your server's IP even with browser bypass enabled. Cloud/datacenter IPs are often flagged as high-risk.

  **Solution:** Combine `--browser` with residential proxies:

  ```bash theme={null}
  ./scrapai crawl spider --project proj --proxy-type residential --browser
  ```

  See [Proxy Configuration](/configuration/proxies#running-on-vpscloud-servers) for details.
</Warning>

## Display Requirements

<Note>
  Cloudflare bypass requires a **visible browser** (not headless). Cloudflare detects and blocks headless browsers.
</Note>

**Platform support:**

* **Windows:** Uses native display automatically ✓
* **macOS:** Uses native display automatically ✓
* **Linux desktop:** Uses native display automatically ✓
* **Linux servers (VPS without GUI):** Auto-detects missing display and uses **Xvfb** (virtual display) ✓

**Installing Xvfb on Linux servers:**

<CodeGroup>
  ```bash Debian/Ubuntu theme={null}
  sudo apt-get install xvfb
  ```

  ```bash RHEL/CentOS theme={null}
  sudo yum install xorg-x11-server-Xvfb
  ```
</CodeGroup>

The crawler automatically detects your environment and uses Xvfb when no display is available on Linux.

## Inspector Usage

<Steps>
  <Step title="Start with default HTTP (fast)">
    Works for most sites:

    ```bash theme={null}
    ./scrapai inspect https://example.com --project proj
    ```
  </Step>

  <Step title="Try browser mode if JS-rendered">
    For JavaScript-heavy sites:

    ```bash theme={null}
    ./scrapai inspect https://example.com --project proj --browser
    ```
  </Step>

  <Step title="Use Cloudflare bypass only when blocked">
    For Cloudflare-protected sites:

    ```bash theme={null}
    ./scrapai inspect https://example.com --project proj --browser
    ```
  </Step>
</Steps>

## Strategies

### Hybrid Mode (Recommended)

Verify the Cloudflare challenge once, then serve everything else over fast HTTP with the cached cookie. **20-100x faster** than browser-only mode.

**How verification works (reactive hold-and-verify — no timer):**

1. The first request to a host has no cookie, so it verifies once. All other requests for that host hold at a single gate while ONE verification runs — the browser is never driven by concurrent requests at the same time.
2. Once a cookie exists, requests go out over HTTP with it (via `curl_cffi` TLS impersonation).
3. If an HTTP response comes back blocked, requests hold again, ONE request re-verifies, and everyone retries with the fresh cookie. There is **no time-based refresh** — cookies are re-fetched only when missing or when a response is actually blocked. If a response is still blocked immediately after a fresh verify, that's treated as a real block (IP/rate limit) and surfaced, not retried forever.

<Note>
  Cookies are cached **per spider + host**. Each hostname (`www.`, `hemeroteca.`, an API subdomain, …) verifies and caches independently — a cookie for one subdomain is never reused for another.
</Note>

<Note>
  Cloudflare verification always routes through the **shared browser service** — one warm browser shared across all crawls, so the challenge is solved once and reused instead of every crawl spawning its own Chrome. If the service isn't running it is started automatically; if it can't be reached the request fails and Scrapy retries it. See the [Browser Service](/guides/browser-service) guide.
</Note>

<Note>
  Do NOT set `CONCURRENT_REQUESTS` - uses Scrapy default of 16 for optimal performance.
</Note>

```json spider.json theme={null}
{
  "settings": {
    "CLOUDFLARE_ENABLED": true,
    "CLOUDFLARE_STRATEGY": "hybrid"
  }
}
```

### Browser-Only Mode (Legacy)

<Warning>
  **Much slower** - uses browser for every request. Requires `CONCURRENT_REQUESTS: 1` to prevent browser conflicts.
</Warning>

```json spider.json theme={null}
{
  "settings": {
    "CLOUDFLARE_ENABLED": true,
    "CLOUDFLARE_STRATEGY": "browser_only",
    "CONCURRENT_REQUESTS": 1
  }
}
```

## Settings Reference

| Setting               | Default  | Description                                                                                                  |
| --------------------- | -------- | ------------------------------------------------------------------------------------------------------------ |
| `CLOUDFLARE_ENABLED`  | false    | Enable browser mode (Cloudflare intent)                                                                      |
| `BROWSER_ENABLED`     | false    | Enable browser mode for plain JS-render sites; alias of `CLOUDFLARE_ENABLED`                                 |
| `CURL_CFFI_ENABLED`   | false    | Use `curl_cffi` TLS impersonation instead of the browser (try first; takes precedence over browser settings) |
| `CLOUDFLARE_STRATEGY` | "hybrid" | "hybrid" (fast) or "browser\_only" (slow)                                                                    |
| `CLOUDFLARE_HEADLESS` | false    | Run browser headless (true = no GUI, worse stealth)                                                          |
| `CF_POST_DELAY`       | 5        | Seconds to wait after a successful verification                                                              |
| `CF_WAIT_SELECTOR`    | —        | CSS selector to wait for before extracting (browser fetch)                                                   |
| `CF_WAIT_TIMEOUT`     | 10       | Max seconds to wait for `CF_WAIT_SELECTOR`                                                                   |
| `CF_MAX_RETRIES`      | 5        | Kept for API compatibility (not used by CloakBrowser)                                                        |
| `CF_RETRY_INTERVAL`   | 1        | Kept for API compatibility (not used by CloakBrowser)                                                        |
| `CONCURRENT_REQUESTS` | 16       | Must be 1 for browser-only mode                                                                              |

## Complete Spider Example

```json spider.json theme={null}
{
  "name": "mysite",
  "allowed_domains": ["example.com"],
  "start_urls": ["https://www.example.com/articles"],
  "rules": [
    {
      "allow": ["/article/[^/]+$"],
      "callback": "parse_article",
      "follow": false,
      "priority": 100
    },
    {
      "allow": ["/articles/"],
      "callback": null,
      "follow": true,
      "priority": 50
    }
  ],
  "settings": {
    "CLOUDFLARE_ENABLED": true,
    "CLOUDFLARE_STRATEGY": "hybrid",
    "CF_POST_DELAY": 5,
    "CF_WAIT_SELECTOR": "h1.title-med-1",
    "DOWNLOAD_DELAY": 2
  }
}
```

## Timeouts & Hang Prevention

<Note>
  **Browser operation timeout:** 300 seconds (5 minutes) per operation to prevent infinite hangs.
</Note>

**Typical operation times:**

* CF verification: 10-60 seconds
* Page load: 5-30 seconds
* Re-verify after a blocked response: 10-30 seconds

<Warning>
  If you consistently hit the 300s timeout, investigate:

  * Network connectivity issues
  * Site blocking your IP/region
  * Browser/Chrome subprocess problems
  * System resource constraints (CPU/memory)
</Warning>

## Troubleshooting

### Crawl Hangs at "Getting/refreshing CF cookies"

**Symptoms:** Browser opens but never navigates.

**Solutions:**

<Steps>
  <Step title="Update to latest version">
    Ensure you're on latest version with timeout fix
  </Step>

  <Step title="Verify browser opens">
    Check browser actually opens (not headless failing)
  </Step>

  <Step title="Check display (Linux servers)">
    Verify Xvfb is installed: `sudo apt-get install xvfb`
  </Step>

  <Step title="Test with inspector">
    Test with `--browser` flag on inspector first:

    ```bash theme={null}
    ./scrapai inspect https://example.com --project proj --browser
    ```
  </Step>

  <Step title="Check system resources">
    Verify CPU, memory, and disk space availability
  </Step>
</Steps>

### Works on One Machine But Not Another

**Debugging steps:**

<Steps>
  <Step title="Test inspector on both machines">
    ```bash theme={null}
    ./scrapai inspect https://example.com --project proj --browser
    ```
  </Step>

  <Step title="Check Chrome installation">
    ```bash theme={null}
    google-chrome --version
    ```
  </Step>

  <Step title="Verify display (Linux)">
    ```bash theme={null}
    echo $DISPLAY  # Should show :99 with Xvfb
    ```
  </Step>

  <Step title="Review logs for errors">
    Check logs for specific error messages
  </Step>

  <Step title="Try different strategy">
    Switch between hybrid and browser\_only modes
  </Step>
</Steps>

### Diagnosing via Logs

**Hybrid mode indicators:**

<CodeGroup>
  ```log Success theme={null}
  [spider|host] Cached N cookies: cf_clearance, ...
  // Cookie cached for this host, HTTP requests working
  ```

  ```log Warning theme={null}
  [spider] Blocked on host - holding to reverify
  // A response came back blocked; all requests hold while ONE re-verifies
  ```

  ```log Error theme={null}
  Still blocked after reverify: https://...
  // Blocked immediately after a fresh cookie => real block (IP/rate limit)
  ```
</CodeGroup>

**Browser-only mode indicators:**

```log theme={null}
Cloudflare verified successfully
Opened persistent browser
Closed browser
// Normal lifecycle
```

### Title Contamination

<Tip>
  If extracted titles show wrong text, set `CF_WAIT_SELECTOR` to the main title element to capture HTML before related content loads.
</Tip>

```json theme={null}
{
  "settings": {
    "CF_WAIT_SELECTOR": "h1.article-title"
  }
}
```

## Related Guides

<CardGroup cols={2}>
  <Card title="Browser Service" icon="server" href="/guides/browser-service">
    Shared warm browser that solves CF once per site
  </Card>

  <Card title="Proxy Escalation" icon="network-wired" href="/guides/proxy-escalation">
    Combine with smart proxy usage
  </Card>

  <Card title="Checkpoint Resume" icon="save" href="/guides/checkpoint-resume">
    Pause and resume long crawls
  </Card>
</CardGroup>