> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Smart Proxy Escalation

> Minimize costs with intelligent proxy usage and expert-in-the-loop escalation

scrapai includes **SmartProxyMiddleware** that intelligently manages proxy usage to avoid blocks while minimizing costs.

## How It Works

SmartProxyMiddleware uses intelligent escalation:

1. **Direct connections first** - Fast and free
2. **Detect blocking** - Automatic detection of 403/429 errors
3. **Datacenter proxy fallback** - Cheap, fast option
4. **Learn domain patterns** - Remembers which domains need proxies
5. **Expert-in-the-loop** - Asks before using expensive residential proxies

<Note>
  **Smart cost control:** Direct connections when possible, datacenter proxies for blocks, residential proxies only with explicit approval. Reduces proxy costs by 80-90%.
</Note>

## Setup

Add proxy credentials to `.env`:

<CodeGroup>
  ```bash Datacenter Proxy theme={null}
  # Datacenter Proxy (default - used with --proxy-type datacenter or no flag)
  DATACENTER_PROXY_USERNAME=your_username
  DATACENTER_PROXY_PASSWORD=your_password
  DATACENTER_PROXY_HOST=dc.yourproxy.com
  DATACENTER_PROXY_PORT=10000
  ```

  ```bash Residential Proxy theme={null}
  # Residential Proxy (used with --proxy-type residential flag)
  RESIDENTIAL_PROXY_USERNAME=your_username
  RESIDENTIAL_PROXY_PASSWORD=your_password
  RESIDENTIAL_PROXY_HOST=residential.yourproxy.com
  RESIDENTIAL_PROXY_PORT=8000
  ```
</CodeGroup>

<Tip>
  **Proxy Configuration:**

  Check your proxy provider's documentation for:

  * Correct hostname and port for datacenter vs residential proxies
  * Rotating vs sticky IP options
  * Authentication requirements

  Since SmartProxyMiddleware uses a single proxy connection, **use rotating IP ports when available** for better IP distribution.
</Tip>

## Usage

### Auto Mode (Default) - Expert-in-the-Loop

```bash theme={null}
# Auto mode (default) - smart escalation
./scrapai crawl spider_name --project proj --limit 10

# Explicit auto mode
./scrapai crawl spider_name --project proj --limit 10 --proxy-type auto
```

When datacenter proxy fails, you'll see:

```
⚠️  EXPERT-IN-THE-LOOP: Datacenter proxy failed for some domains
🏠 Residential proxy is available but may incur HIGHER COSTS

Blocked domains: example.com, site.org

To proceed with residential proxy, run:
  ./scrapai crawl spider_name --project proj --proxy-type residential
```

<Warning>
  **Cost protection:** Residential proxies require explicit user approval - no surprise costs!
</Warning>

### Explicit Proxy Modes

<CodeGroup>
  ```bash Datacenter Only theme={null}
  # Force datacenter proxy only (even if residential configured)
  ./scrapai crawl spider_name --project proj --limit 10 --proxy-type datacenter
  ```

  ```bash Residential Only theme={null}
  # Force residential proxy (explicit approval given)
  ./scrapai crawl spider_name --project proj --limit 10 --proxy-type residential
  ```
</CodeGroup>

## Configuration

<Tip>
  **No spider configuration needed!** SmartProxyMiddleware works automatically for all spiders once configured in `.env`.
</Tip>

The middleware is enabled by default in `settings.py` with priority 350.

## Statistics Tracking

SmartProxyMiddleware tracks detailed usage statistics:

* **Direct requests** - Connections without proxy
* **Proxy requests** - Connections using proxy
* **Blocked retries** - Requests that hit 403/429 and retried with proxy
* **Blocked domains** - Domains that consistently need proxies

**Statistics logged when spider closes:**

```log theme={null}
📊 Proxy Statistics for 'spider_name':
   Direct requests: 1847
   Proxy requests: 153
   Blocked & retried: 153
   Blocked domains: 2
   Domains that needed proxy: example.com, protected-site.com
```

## Proxy Providers

SmartProxyMiddleware works with any HTTP proxy provider. Popular options include Bright Data, Oxylabs, IPRoyal, Smartproxy, and others.

## Troubleshooting

### Proxy Not Being Used

<Steps>
  <Step title="Check .env configuration">
    Verify all 4 variables are set (USERNAME, PASSWORD, HOST, PORT)
  </Step>

  <Step title="Verify credentials">
    Ensure proxy credentials are correct
  </Step>

  <Step title="Test proxy manually">
    ```bash theme={null}
    curl -x http://user:pass@host:port https://httpbin.org/ip
    ```
  </Step>

  <Step title="Check logs">
    Look for "Datacenter proxy available" message on spider start
  </Step>
</Steps>

### Still Getting Blocked with Proxy

<Steps>
  <Step title="Check if proxy IP is blocked">
    Proxy IP may already be blocked by target site
  </Step>

  <Step title="Try different provider">
    Switch to alternative proxy provider
  </Step>

  <Step title="Add delays">
    Increase `DOWNLOAD_DELAY` in spider config:

    ```json theme={null}
    {"settings": {"DOWNLOAD_DELAY": 2}}
    ```
  </Step>

  <Step title="Reduce concurrency">
    Lower `CONCURRENT_REQUESTS` in spider config:

    ```json theme={null}
    {"settings": {"CONCURRENT_REQUESTS": 8}}
    ```
  </Step>
</Steps>

### Proxy Costs Too High

<Note>
  SmartProxyMiddleware should already minimize costs by using direct connections first.
</Note>

If costs are still high:

1. Check which domains are marked as blocked (in stats at spider close)
2. Verify those domains actually need proxies
3. Consider if site has changed and unblocking is possible
4. Some sites may require proxies for all requests - this is expected

## Related Guides

<CardGroup cols={2}>
  <Card title="Cloudflare Bypass" icon="shield" href="/guides/cloudflare-bypass">
    Handle Cloudflare-protected sites
  </Card>

  <Card title="Checkpoint Resume" icon="save" href="/guides/checkpoint-resume">
    Pause and resume long crawls
  </Card>
</CardGroup>