> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Installation

> Install scrapai CLI on Linux, macOS, or Windows

## System Requirements

<CardGroup cols={3}>
  <Card title="Python" icon="python">
    Version **3.9** or higher
  </Card>

  <Card title="Git" icon="git">
    For cloning the repository
  </Card>

  <Card title="Disk Space" icon="hard-drive">
    \~500 MB (dependencies + browser)
  </Card>
</CardGroup>

<Note>
  scrapai uses SQLite by default (no database installation required). For production scale, PostgreSQL is recommended.
</Note>

## Supported Platforms

* **Linux** (Ubuntu, Debian, CentOS, Fedora, Arch)
* **macOS** (Intel and Apple Silicon)
* **Windows** (10/11 via WSL only)

## Installation Steps

<Steps>
  <Step title="Install Python 3.9+">
    <Tabs>
      <Tab title="Linux">
        <Accordion title="Ubuntu/Debian">
          ```bash theme={null}
          sudo apt update
          sudo apt install python3.9 python3.9-venv python3-pip git
          ```
        </Accordion>

        <Accordion title="CentOS/Fedora">
          ```bash theme={null}
          sudo dnf install python39 python39-pip git
          ```
        </Accordion>

        <Accordion title="Arch Linux">
          ```bash theme={null}
          sudo pacman -S python python-pip git
          ```
        </Accordion>

        Verify installation:

        ```bash theme={null}
        python3 --version
        ```
      </Tab>

      <Tab title="macOS">
        Install via Homebrew:

        ```bash theme={null}
        brew install python@3.9 git
        ```

        Or download from [python.org](https://www.python.org/downloads/macos/)

        Verify installation:

        ```bash theme={null}
        python3 --version
        ```
      </Tab>

      <Tab title="Windows">
        **Windows users must use WSL (Windows Subsystem for Linux).**

        [Install WSL](https://learn.microsoft.com/en-us/windows/wsl/install), then follow the Linux instructions above.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Clone the repository">
    ```bash theme={null}
    git clone https://github.com/discourselab/scrapai-cli.git
    cd scrapai-cli
    ```

    <Tip>
      Clone to a location with write permissions. Avoid system directories like `/usr/local/`.
    </Tip>
  </Step>

  <Step title="Run setup">
    ```bash theme={null}
    ./scrapai setup
    ```

    The setup process will:

    <Steps>
      <Step title="Create virtual environment">
        Creates a `.venv` directory with isolated Python packages
      </Step>

      <Step title="Install dependencies">
        Installs Scrapy, SQLAlchemy, Alembic, newspaper4k, trafilatura, Playwright, and more
      </Step>

      <Step title="Install Playwright Chromium">
        Downloads Chromium browser for JavaScript rendering and Cloudflare bypass

        <Warning>
          **Linux users**: If Chromium fails to launch later, you may need to install system dependencies:

          ```bash theme={null}
          sudo .venv/bin/python -m playwright install-deps chromium
          ```

          This requires sudo because it installs system packages (fonts, libraries, etc.).
        </Warning>
      </Step>

      <Step title="Create .env file">
        Copies `.env.example` to `.env` with default SQLite configuration
      </Step>

      <Step title="Initialize database">
        Runs Alembic migrations to create the database schema
      </Step>

      <Step title="Configure Claude Code permissions">
        If using AI agents, sets up permission rules in `.claude/settings.local.json`
      </Step>
    </Steps>

    <Accordion title="Expected output">
      ```
      🚀 Setting up scrapai environment...
      📦 Creating virtual environment...
      ✅ Virtual environment created
      📋 Installing requirements...
      ✅ Requirements installed
      🌐 Installing Playwright Chromium browser...
      ✅ Playwright Chromium installed
      📝 Creating .env from .env.example...
      ✅ .env file created (using SQLite by default)
      📁 Checking data directory permissions...
      ✅ Have permission to write to data directory: ./data
      🗄️  Initializing database...
      ✅ Database initialized with migrations
      🔧 Configuring Claude Code permissions...
      ✅ Claude Code permissions configured
      🎉 scrapai setup complete!
      ```
    </Accordion>
  </Step>

  <Step title="Verify installation">
    ```bash theme={null}
    ./scrapai verify
    ```

    You should see:

    ```
    🔍 Verifying scrapai environment...

    ✅ Virtual environment exists
    ✅ Core dependencies installed
    ✅ Database initialized

    🎉 Environment is ready!
    ```

    <Warning>
      If any checks fail, re-run `./scrapai setup`. If issues persist, see [Troubleshooting](#troubleshooting) below.
    </Warning>
  </Step>
</Steps>

## Configuration

### Database Configuration

scrapai uses **SQLite by default** (no setup required). For production, you can transfer your existing data to **PostgreSQL**:

<Steps>
  <Step title="Get PostgreSQL">
    Use a managed service (AWS RDS, DigitalOcean, Supabase) or install locally:

    ```bash theme={null}
    # Linux: sudo apt install postgresql postgresql-contrib
    # macOS: brew install postgresql
    ```
  </Step>

  <Step title="Update .env">
    ```bash theme={null}
    DATABASE_URL=postgresql://user:password@host:5432/scrapai
    ```
  </Step>

  <Step title="Run migrations and transfer">
    ```bash theme={null}
    ./scrapai db migrate
    ./scrapai db transfer sqlite:///scrapai.db
    ```
  </Step>
</Steps>

### Proxy Configuration

scrapai supports smart proxy escalation. Configure proxies in `.env`:

```bash theme={null}
# Datacenter Proxy (recommended - faster, cheaper)
DATACENTER_PROXY_USERNAME=your_username
DATACENTER_PROXY_PASSWORD=your_password
DATACENTER_PROXY_HOST=your-datacenter-proxy.com
DATACENTER_PROXY_PORT=10000  # Port 10000 = rotating IPs

# Residential Proxy (for sites that block datacenter IPs)
RESIDENTIAL_PROXY_USERNAME=your_username
RESIDENTIAL_PROXY_PASSWORD=your_password
RESIDENTIAL_PROXY_HOST=your-residential-proxy.com
RESIDENTIAL_PROXY_PORT=7000  # Port 7000 = rotating residential IPs
```

<Tip>
  Proxies are optional. scrapai starts with direct connections and only uses proxies when needed (403/429 errors). It learns which domains require proxies and remembers for future crawls.
</Tip>

### S3 Storage Configuration

For automatic uploads to S3-compatible storage (Hetzner, DigitalOcean Spaces, Wasabi, Backblaze, etc.):

```bash theme={null}
S3_ACCESS_KEY=your_access_key_here
S3_SECRET_KEY=your_secret_key_here
S3_ENDPOINT=https://your-s3-endpoint.com
S3_BUCKET=your-bucket-name
```

## Troubleshooting

<AccordionGroup>
  <Accordion title="Virtual environment creation fails">
    **Error**: `The virtual environment was not created successfully`

    **Solution**:

    ```bash theme={null}
    # Install venv module
    sudo apt install python3.9-venv  # Ubuntu/Debian

    # Or use a different Python version
    python3.10 -m venv .venv
    ```
  </Accordion>

  <Accordion title="Playwright Chromium won't launch (Linux)">
    **Error**: `Error: browserType.launch: Host system is missing dependencies`

    **Solution**: Install system dependencies:

    ```bash theme={null}
    sudo .venv/bin/python -m playwright install-deps chromium
    ```

    This installs required system packages (fonts, libraries, etc.).
  </Accordion>

  <Accordion title="Permission denied when writing to data directory">
    **Error**: `PermissionError: [Errno 13] Permission denied: './data'`

    **Solution**: Change the data directory in `.env`:

    ```bash theme={null}
    DATA_DIR=~/scrapai-data
    ```

    Or fix permissions:

    ```bash theme={null}
    sudo chown -R $USER:$USER ./data
    chmod -R 755 ./data
    ```
  </Accordion>

  <Accordion title="Database connection fails (PostgreSQL)">
    **Error**: `sqlalchemy.exc.OperationalError: could not connect to server`

    **Solutions**:

    1. Verify PostgreSQL is running:
       ```bash theme={null}
       sudo systemctl status postgresql
       ```

    2. Check your `DATABASE_URL` in `.env`

    3. Test connection:
       ```bash theme={null}
       psql -U scrapai_user -d scrapai -h localhost
       ```

    4. Check PostgreSQL logs:
       ```bash theme={null}
       sudo tail -f /var/log/postgresql/postgresql-*.log
       ```
  </Accordion>

  <Accordion title="Command not found: ./scrapai (Linux/macOS)">
    **Error**: `bash: ./scrapai: No such file or directory`

    **Solution**: Make the script executable:

    ```bash theme={null}
    chmod +x scrapai
    ./scrapai verify
    ```
  </Accordion>

  <Accordion title="Python version too old">
    **Error**: `ERROR: This package requires Python 3.9 or higher`

    **Solution**: Install a newer Python version:

    ```bash theme={null}
    # Ubuntu/Debian
    sudo apt install python3.10 python3.10-venv

    # macOS
    brew install python@3.10

    # Then recreate the venv
    rm -rf .venv
    python3.10 -m venv .venv
    ./scrapai setup
    ```
  </Accordion>
</AccordionGroup>

## Upgrading

To upgrade to the latest version:

```bash theme={null}
git pull origin main
./scrapai setup  # Re-run setup to install new dependencies
./scrapai db migrate  # Apply any new database migrations
```

<Warning>
  Always backup your database before upgrading:

  ```bash theme={null}
  # SQLite
  cp scrapai.db scrapai.db.backup

  # PostgreSQL
  pg_dump -U scrapai_user scrapai > scrapai_backup.sql
  ```
</Warning>

## Uninstallation

To completely remove scrapai:

```bash theme={null}
cd scrapai-cli

# Remove virtual environment
rm -rf .venv

# Remove database (if using SQLite)
rm scrapai.db

# Remove data directory
rm -rf data/

# Remove the repository
cd ..
rm -rf scrapai-cli/
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Quick Start" icon="bolt" href="/quickstart">
    Build your first scraper in 5 minutes
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/cli/overview">
    Complete command reference
  </Card>

  <Card title="Configuration" icon="gear" href="/configuration/environment">
    Advanced configuration options
  </Card>

  <Card title="GitHub Repository" icon="github" href="https://github.com/discourselab/scrapai-cli">
    View source code and report issues
  </Card>
</CardGroup>