Skip to main content
scrapai includes SmartProxyMiddleware that intelligently manages proxy usage to avoid blocks while minimizing costs.

​
How It Works

SmartProxyMiddleware uses intelligent escalation:
  1. Direct connections first - Fast and free
  2. Detect blocking - Automatic detection of 403/429 errors
  3. Datacenter proxy fallback - Cheap, fast option
  4. Learn domain patterns - Remembers which domains need proxies
  5. Expert-in-the-loop - Asks before using expensive residential proxies
Smart cost control: Direct connections when possible, datacenter proxies for blocks, residential proxies only with explicit approval. Reduces proxy costs by 80-90%.

​
Setup

Add proxy credentials to .env:
Proxy Configuration:Check your proxy provider’s documentation for:
  • Correct hostname and port for datacenter vs residential proxies
  • Rotating vs sticky IP options
  • Authentication requirements
Since SmartProxyMiddleware uses a single proxy connection, use rotating IP ports when available for better IP distribution.

​
Usage

​
Auto Mode (Default) - Expert-in-the-Loop

When datacenter proxy fails, you’ll see:
Cost protection: Residential proxies require explicit user approval - no surprise costs!

​
Explicit Proxy Modes

​
Configuration

No spider configuration needed! SmartProxyMiddleware works automatically for all spiders once configured in .env.
The middleware is enabled by default in settings.py with priority 350.

​
Statistics Tracking

SmartProxyMiddleware tracks detailed usage statistics:
  • Direct requests - Connections without proxy
  • Proxy requests - Connections using proxy
  • Blocked retries - Requests that hit 403/429 and retried with proxy
  • Blocked domains - Domains that consistently need proxies
Statistics logged when spider closes:

​
Proxy Providers

SmartProxyMiddleware works with any HTTP proxy provider. Popular options include Bright Data, Oxylabs, IPRoyal, Smartproxy, and others.

​
Troubleshooting

​
Proxy Not Being Used

1

Check .env configuration

Verify all 4 variables are set (USERNAME, PASSWORD, HOST, PORT)
2

Verify credentials

Ensure proxy credentials are correct
3

Test proxy manually

4

Check logs

Look for “Datacenter proxy available” message on spider start

​
Still Getting Blocked with Proxy

1

Check if proxy IP is blocked

Proxy IP may already be blocked by target site
2

Try different provider

Switch to alternative proxy provider
3

Add delays

Increase DOWNLOAD_DELAY in spider config:
4

Reduce concurrency

Lower CONCURRENT_REQUESTS in spider config:

​
Proxy Costs Too High

SmartProxyMiddleware should already minimize costs by using direct connections first.
If costs are still high:
  1. Check which domains are marked as blocked (in stats at spider close)
  2. Verify those domains actually need proxies
  3. Consider if site has changed and unblocking is possible
  4. Some sites may require proxies for all requests - this is expected

Cloudflare Bypass

Handle Cloudflare-protected sites

Checkpoint Resume

Pause and resume long crawls