> ## Documentation Index > Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt > Use this file to discover all available pages before exploring further. # scrapai vs Other Tools > A practical comparison of scrapai, Scrapling, and crawl4ai for anyone choosing a scraping tool A practical comparison for anyone choosing a scraping tool. Each tool is genuinely good at different things. ## Overview | | **Scrapling** | **crawl4ai** | **scrapai** | | ----------------------- | -------------------------------------- | ------------------------------------------------ | ---------------------------------- | | **What it is** | Scraping library with adaptive parsing | Crawler & content extractor with LLM integration | AI-agent-driven scraper management | | **Best for** | Stealth scraping, anti-bot evasion | Exploration, research, prototyping | Managing many scrapers at scale | | **Who writes scrapers** | You (Python) | You (Python) | AI agent (JSON configs) | | **License** | BSD-3 | Apache 2.0 | AGPL-3.0 | **The short version:** Scrapling and crawl4ai are excellent scraping libraries that give developers full control. scrapai is a different thing: an orchestration layer where an AI agent builds and manages scrapers, and you talk to it in English. They're not competing for the same job. ### Codebase Size Measured with [pygount](https://github.com/roskakori/pygount), counting actual code lines only (no blanks, no comments, no docstrings). Tests, examples, and docs excluded for all three. | Repository | Files | Code Lines | Comment Lines | Comment % | | ------------- | ----- | ---------- | ------------- | --------- | | **scrapai** | 37 | 4,028 | 895 | 14% | | **Scrapling** | 43 | 5,875 | 2,063 | 21% | | **crawl4ai** | 87 | 26,850 | 10,326 | 21% | scrapai is the smallest codebase. crawl4ai is \~7x larger, which reflects its broader scope (markdown conversion, LLM integration, media extraction, Docker API). Scrapling is closer in size but includes more inline documentation. ## Getting Started | | Scrapling | crawl4ai | scrapai | | ------------------------- | -------------------------------------------------- | ----------------------------------------- | --------------------------------------- | | **Install** | `pip install scrapling[all]` + `scrapling install` | `pip install crawl4ai` + `crawl4ai-setup` | `git clone` + `./scrapai setup` | | **Time to first scraper** | 30-60 min (inspect, write code, test, debug) | 30-60 min (same) | \~5 min (setup, launch agent, give URL) | | **Skills needed** | Python, CSS/XPath, HTTP | Python, async/await, CSS/XPath | Plain English | With Scrapling or crawl4ai, you're writing Python. With scrapai, the agent handles analysis and config generation, but developers can also hand-write, edit, or override any config. The difference is where you spend your time: writing extraction code vs reviewing and refining what the agent produces. ## Economics ### Building Scrapers | | Scrapling / crawl4ai | scrapai | | ----------------------- | --------------------------- | ------------------------------- | | **1 scraper** | 30-60 min developer time | 3-6 min, \~\$1-3 in API tokens | | **100 scrapers** | 50-100 hours developer time | \~\$100-300 in tokens, 1-2 days | | **Adding scraper #101** | Write another Python script | "Add this URL to my project" | ### Running Scrapers This is where the approaches really diverge. | | Scrapling | crawl4ai | scrapai | | ----------------- | --------- | ---------------------------------------------------------- | ------------------------ | | **AI at runtime** | No | Optional (two modes) | No, deterministic Scrapy | | **Per-page cost** | \$0 | \$0 with cached schemas, LLM cost with per-page extraction | \$0 | crawl4ai has two extraction modes. `LLMExtractionStrategy` calls the LLM on every page, which is powerful for exploration but expensive at scale. `JsonCssExtractionStrategy` uses `generate_schema()` to call the LLM once, generate CSS/XPath selectors, cache the schema as a JSON file, and reuse it for all subsequent pages with no LLM calls. That second mode follows the same principle as scrapai: AI once, deterministic forever. The difference is scope. crawl4ai's cached schemas cover extraction selectors for one page type. scrapai's spider configs cover the full pipeline: URL discovery rules, extraction settings, Cloudflare config, proxy behavior, and crawl parameters, all stored in a database and managed through a CLI. crawl4ai gives you the extraction building block; scrapai wraps the entire workflow. ### Maintenance When a site redesigns and breaks a scraper: | | Scrapling / crawl4ai | scrapai | | --------------- | ------------------------------------ | ---------------------------------------------------------- | | **Detection** | You notice it's broken | [AI-assisted test crawls](/guides/ai-assisted-maintenance) | | **Fix** | Developer investigates, updates code | Agent re-analyzes, updates config | | **Time to fix** | 30-60 min | 3-6 min, \~\$1-3 in tokens | Scrapling has a unique advantage here: its adaptive parser can relocate elements after site redesigns, potentially surviving changes that would break other tools. ## Anti-Bot & Cloudflare ### Stealth Capabilities | | Scrapling | crawl4ai | scrapai | | ---------------------- | ------------------------------- | ------------------------------- | ---------------------------------------------------------------------------- | | **Browser engine** | Patchright (patched Playwright) | Patchright (patched Playwright) | [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) (16 C++ patches) | | **reCAPTCHA v3 score** | Not documented | Not documented | **0.9 (verified)** | | **Detection bypass** | Cloudflare, most sites | Cloudflare + Capsolver | **Cloudflare Turnstile, FingerprintJS, BrowserScan, DataDome (30/30 tests)** | ### Cloudflare Challenges | | Scrapling | crawl4ai | scrapai | | ---------------------------------------- | ----------------------- | --------------------- | ----------------------------------------------- | | **Non-interactive** ("Just a moment...") | Waits for title change | Stealth handles it | **Auto-solved (CloakBrowser 0.9 score)** | | **Turnstile (managed)** | Auto-clicks with offset | Relies on stealth | **Auto-solved or single click** | | **Turnstile (non-interactive)** | Depends on stealth | Depends on stealth | **Auto-passed (no interaction)** | | **reCAPTCHA v3** | Not documented | Not documented | **0.9 score (human-level, verified)** | | **CAPTCHA solving** | Auto-click only | Capsolver integration | **Auto-solved via stealth (no service needed)** | ### Speed on Cloudflare-Protected Sites This is where architecture matters more than stealth. | | Scrapling (StealthySession) | crawl4ai | scrapai (CloakBrowser hybrid) | | ----------------------- | ----------------------------- | ------------------------------------- | -------------------------------------------------------------- | | **CF solution** | Solved once per session | Per request (unless stealth bypasses) | **CloakBrowser solves once, cookies cached \~10 min** | | **Subsequent requests** | Browser per request (\~5-10s) | Browser per request (\~5-10s) | **HTTP with cached cookies (\~0.1-0.5s)** | | **Browser lifecycle** | Stays open for session | Stays open for session | **Shuts down after cookie extraction, reopens every \~10 min** | | **Concurrency** | Limited by browser tabs | Limited by browser tabs | **16+ concurrent HTTP requests** | | **100 pages** | \~8-16 min | \~8-16 min | **\~1 min** | | **1,000 pages** | \~1.5-3 hours | \~1.5-3 hours | **\~8 min** | Scrapling's `StealthySession` persists cookies but keeps the browser open for all requests (\~5-10s each). scrapai solves once, caches cookies, shuts down browser, then uses fast HTTP (\~0.1-0.5s per page). ## Data Extraction | | Scrapling | crawl4ai | scrapai | | -------------------- | ------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------- | | **Approach** | Your CSS/XPath selectors | Full-page markdown, per-page LLM, or LLM-generated cached schemas | Targeted field extraction (newspaper/trafilatura or custom callbacks) | | **Adaptive parsing** | Yes (survives redesigns) | No | No | | **LLM integration** | MCP server | Native (per-page) | At build time only | | **Token efficiency** | Depends on your selectors | \~6,300 tokens per article (BBC example) | \~1,200 tokens (same article) | crawl4ai returns markdown of the full page, including navigation, sidebars, and footers. Heuristic filters help but aren't perfect. Great for exploration ("I don't know what I'm looking for yet") but expensive at scale if you're feeding results to an LLM. scrapai extracts only the fields you need. For a BBC article: title, content, author, date. Clean structured data, 5x fewer tokens than full-page markdown. For non-article content (products, jobs, listings), the agent writes custom callbacks with field-level selectors. Scrapling's adaptive parser is unique: it tracks elements across site redesigns using fuzzy matching. Neither crawl4ai nor scrapai have anything like this. ## Production Infrastructure | Feature | Scrapling | crawl4ai | scrapai | | --------------------- | -------------- | ---------------------- | ---------------------------------------------------------------------- | | **Pause/resume** | crawldir-based | Crash recovery | Scrapy JOBDIR | | **Proxy management** | ProxyRotator | proxy\_config | Smart escalation (direct → datacenter auto, residential with approval) | | **Incremental crawl** | No | No | DeltaFetch (skip already-scraped URLs) | | **Queue system** | No | No | Database-backed with priorities | | **Export formats** | JSON/JSONL | Markdown, JSON | CSV, JSON, JSONL, Parquet | | **Scheduling** | No | Docker + external cron | Cron + Pueue queue | | **S3 upload** | No | No | Built-in | scrapai has more production features because that's what it was built for: running many scrapers on a schedule. Scrapling and crawl4ai are libraries, not platforms. You'd build these features yourself on top of them (and many teams do). ## Teams and Onboarding **With Scrapling / crawl4ai:** Team members need Python, async programming, and CSS/XPath skills. Changing settings across many scrapers means editing many files. **With scrapai:** Natural language interface, configs stored in database. Changing settings across 100 scrapers is one database query. ## AI Agents + Scraping: The Security Question AI agents are already being paired with scraping libraries. [OpenClaw users are combining it with Scrapling](https://www.wired.com/story/openclaw-users-bypass-anti-bot-systems-cloudflare-scrapling) to build autonomous scraping pipelines where the agent writes Python code, executes it, and downloads data. The combination is powerful, but the risks are real. In February 2026, an [OpenClaw agent deleted 200+ emails](https://techcrunch.com/2026/02/23/a-meta-ai-security-researcher-said-an-openclaw-agent-ran-amok-on-her-inbox/) after context compaction caused it to lose safety constraints, and [30,000+ instances were found exposed](https://www.bitsight.com/blog/openclaw-ai-security-risks-exposed-instances) with leaked credentials. When an agent writes and executes arbitrary code while processing content from untrusted websites, prompt injection and context compaction become real attack surfaces. There are two fundamentally different approaches: **Approach A: AI writes code and executes it.** The agent generates Python, runs it on the host or in a container, and returns results. The agent has full programmatic power. If it hallucinates or encounters a prompt injection, the blast radius is whatever the agent has access to. **Approach B: AI writes config, executes predefined commands.** The agent produces structured data (JSON configs) and interacts through a defined CLI. It never writes executable code. At runtime, a deterministic engine (Scrapy) loads the config. The worst case is a bad config that extracts wrong fields, caught during testing. | | Approach A: AI writes code | Approach B: AI writes config | | --------------------------- | ------------------------------------------ | -------------------------------------- | | **Example** | OpenClaw + Scrapling | scrapai | | **What AI produces** | Arbitrary Python | JSON configs | | **Runtime** | AI-generated code executes | Deterministic engine loads config | | **AI at runtime?** | Yes | No, AI only at build time | | **Blast radius of error** | Arbitrary code execution | Bad config, bad scraper | | **Prompt injection risk** | High (malicious page → code execution) | Low (malicious page → bad config data) | | **Context compaction risk** | Safety constraints can be silently dropped | No AI at runtime, not applicable | Scrapling and crawl4ai as standalone libraries avoid the agent risk entirely: developers write the code, review it, and control execution. The risk only appears when you pair them with an autonomous agent framework, where the security model shifts from "developer in the loop" to "AI writes and runs code unsupervised." Neither approach is universally right. Writing code is more flexible. Writing config is more predictable. For scraping at scale, where you're processing hundreds of untrusted websites, predictability matters more than flexibility. ## Agent Compatibility scrapai works with two categories of agents: **coding agents** (interactive, developer in the loop) and **Claws** (autonomous runtimes, no developer in the loop). ### Coding Agents **Claude Code** is our primary development and testing environment. [`CLAUDE.md`](https://github.com/discourselab/scrapai-cli/blob/main/CLAUDE.md) contains the complete workflow, and `./scrapai setup` configures permission rules (allow/deny lists) that block Python modification at the tool level. **Other coding agents** (OpenCode, Cursor, Antigravity, etc.) should work. `Agents.md` provides the same workflow instructions, but these agents don't have Claude Code's permission enforcement. Review changes carefully. ### Claws (Autonomous Agent Runtimes) Claws are headless agent runtimes triggered from Telegram bots, APIs, or scheduled tasks. No developer sitting in a terminal. We tested with [NanoClaw](https://github.com/qwibitai/nanoclaw) because it aligns with what we think matters for autonomous scraping: * **Lightweight.** Minimalist by design, small codebase, small attack surface. * **Container isolation.** Agents run in isolated containers with explicit resource limits. Containerization doesn't solve everything, but it limits the blast radius. * **Two layers of protection.** NanoClaw's container isolation + scrapai's config-only architecture means even a compromised agent can only produce JSON configs and run predefined CLI commands, inside a sandbox. Our initial test: NanoClaw running from a Telegram bot, reading scrapai's workflow, producing working spider configs. More rigorous testing is in progress, particularly around Cloudflare-protected sites and error recovery. scrapai should work with any Claw runtime that can read instructions and execute shell commands. ### Enforcement With Claude Code, permission rules block `Write(**/*.py)`, `Edit(**/*.py)`, and destructive shell commands. This is hard enforcement at the tool level. With other coding agents and Claws, the workflow design (JSON configs + CLI boundary) makes code modification the unnatural path, but there's no hard enforcement. The config-only architecture provides a meaningful security layer regardless of agent, but only Claude Code guarantees the agent can't sidestep it. ## Deployment | | Scrapling | crawl4ai | scrapai | | -------------------- | -------------- | -------------------- | ------------------------------ | | **Headless servers** | Supported | Supported | Supported (Xvfb auto-detected) | | **Docker** | Official image | Official image + API | No official image | scrapai auto-detects missing displays on Linux servers and sets up Xvfb for Cloudflare verification. The virtual display is only needed \~once per 10 min, then everything switches to HTTP. ## Community & Maturity | | Scrapling | crawl4ai | scrapai | | ----------------- | ------------ | --------------------- | --------- | | **GitHub stars** | Growing fast | Large community | Small/new | | **Docker image** | Official | Official + playground | No | | **MCP server** | Yes | Yes | No | | **First release** | 2024 | 2024 | 2025 | crawl4ai has the largest community by far. Scrapling is newer but growing fast. scrapai has been production-tested across hundreds of websites. The public community is just starting. ## Which Tool When? | Scenario | Best pick | Why | | --------------------------------------- | --------------------- | ----------------------------------------------------- | | **One-off scrape, full control** | Scrapling or crawl4ai | Developer tools for developers | | **Exploring an unfamiliar site** | crawl4ai | Returns everything in one call | | **Hard CAPTCHAs** | crawl4ai | Capsolver integration | | **Site changes layout often** | Scrapling or scrapai | Scrapling: adaptive parser. scrapai: agent auto-fixes | | **Highest reCAPTCHA score** | scrapai | CloakBrowser (0.9 vs config-level tools) | | **Non-technical team** | scrapai | Plain English, no code | | **Managing 50+ scrapers** | scrapai | Database + agent + queue | | **Cloudflare site, thousands of pages** | scrapai | Cookie caching = 10x+ faster | | **Team with turnover** | scrapai | No knowledge transfer problem | | **Feeding data to LLMs** | scrapai | 5x fewer tokens than markdown | | **Research / prototyping** | crawl4ai | Rich results, LLM integration | ## Summary **Scrapling** is the best scraping library for stealth and control. BrowserForge fingerprints, 55 browser flags, adaptive parsing that survives site redesigns. If you're a developer who wants to write Python and have maximum anti-detection, this is it. **crawl4ai** is the best exploration and research tool. Point it at a site, get back HTML, markdown, media, links, screenshots, metadata, everything in one call. Largest community, Capsolver for CAPTCHAs, Docker deployment with monitoring. The "I don't know what I'm looking for yet" tool. **scrapai** is built for managing scrapers at scale. An AI agent builds them, a database stores them, Scrapy runs them. CloakBrowser (0.9 reCAPTCHA), cookie-cached Cloudflare bypass, smart proxy escalation, queue system, automated health checks. We built it because we needed to scrape hundreds of sites and couldn't staff a team to write individual scrapers. Trade-off: you depend on AI agent quality and pay token costs instead of developer hours. They're different tools for different problems. At small scale, pick whichever fits how you think. At large scale, the question shifts from "which library has better selectors" to "how do I manage hundreds of scrapers without a dedicated team." *** *Based on codebase analysis of all three projects, February 2026.*