Setup & Verification

setup

Install virtual environment, dependencies, and initialize database.

Syntax

./scrapai setup [--skip-deps]

Options

--skip-deps

flag

Skip dependency installation (useful when re-running setup after manual changes)

What It Does

Creates virtual environment at .venv/ (skips if exists)
Installs Python dependencies from requirements.txt
Installs Playwright Chromium for browser automation
Creates .env file from .env.example (if missing)
Tests data directory permissions by writing a test file
Runs database migrations via Alembic
Configures Claude Code permissions (if using Claude Code agent)

Output

$ ./scrapai setup
🚀 Setting up scrapai environment...
📦 Creating virtual environment...
✅ Virtual environment created
📋 Installing requirements...
✅ Requirements installed
🌐 Installing Playwright Chromium browser...
✅ Playwright Chromium installed
📝 Creating .env from .env.example...
✅ .env file created (using SQLite by default)
📁 Checking data directory permissions...
✅ Have permission to write to data directory: ./data
🗄️  Initializing database...
✅ Database initialized with migrations
🔧 Configuring Claude Code permissions...
✅ Claude Code permissions configured
🎉 scrapai setup complete!
📝 You can now:
   • List spiders: ./scrapai spiders list --project <name>
   • Import spiders: ./scrapai spiders import <file> --project <name>
   • Run crawls: ./scrapai crawl <spider_name> --project <name>

Platform Notes

Linux

Playwright Chromium requires system dependencies. If browser fails to launch:

sudo .venv/bin/python -m playwright install-deps chromium

This command requires sudo as it installs system packages (libglib, libnss3, etc.).

Windows

Use scrapai or scrapai.bat instead of ./scrapai:

scrapai setup

Skip Dependencies

Runs migrations and permission checks without reinstalling packages:

./scrapai setup --skip-deps

verify

Verify environment setup without installing anything. Useful for troubleshooting or CI/CD pipelines.

Syntax

./scrapai verify

What It Checks

Virtual environment exists at .venv/
Core dependencies installed (scrapy, sqlalchemy, alembic)
Database initialized (checks current Alembic revision)

Output

Success

$ ./scrapai verify
🔍 Verifying scrapai environment...

✅ Virtual environment exists
✅ Core dependencies installed
✅ Database initialized

🎉 Environment is ready!
📝 You can now:
   • List spiders: ./scrapai spiders list --project <name>
   • Import spiders: ./scrapai spiders import <file> --project <name>
   • Run crawls: ./scrapai crawl <spider_name> --project <name>

Missing Setup

$ ./scrapai verify
🔍 Verifying scrapai environment...

❌ Virtual environment not found
   Run: ./scrapai setup

⚠️  Environment setup incomplete
   Run: ./scrapai setup

Claude Code Permissions

Setup configures Claude Code agent permissions in .claude/settings.local.json.

Allow List

[
  "Read",
  "Write",
  "Edit",
  "Update",
  "Glob",
  "Grep",
  "Bash(./scrapai:*)",
  "Bash(source:*)",
  "Bash(sqlite3:*)",
  "Bash(psql:*)",
  "Bash(xvfb-run:*)"
]

Deny List

[
  "Edit(scrapai)",
  "Update(scrapai)",
  "Edit(.claude/*)",
  "Update(.claude/*)",
  "Write(**/*.py)",
  "Edit(**/*.py)",
  "Update(**/*.py)",
  "MultiEdit(**/*.py)",
  "Write(.env)",
  "Write(secrets/**)",
  "Write(config/**/*.key)",
  "Write(**/*password*)",
  "Write(**/*secret*)",
  "WebFetch",
  "WebSearch",
  "Bash(rm:*)"
]

These permissions ensure the agent writes config (JSON), not code (Python). This is a core security principle of scrapai’s agent safety model.

Environment Variables

The .env file created by setup:

# Data directory (default: ./data)
DATA_DIR=./data

# Database (default: SQLite)
DATABASE_URL=sqlite:///scrapai.db

# For PostgreSQL:
# DATABASE_URL=postgresql://user:password@localhost:5432/scrapai

# Proxy settings (optional)
DATACENTER_PROXY_USERNAME=
DATACENTER_PROXY_PASSWORD=
DATACENTER_PROXY_HOST=
DATACENTER_PROXY_PORT=

RESIDENTIAL_PROXY_USERNAME=
RESIDENTIAL_PROXY_PASSWORD=
RESIDENTIAL_PROXY_HOST=
RESIDENTIAL_PROXY_PORT=

# S3 storage (optional, auto-upload on production crawls)
S3_ENDPOINT=
S3_BUCKET=

Troubleshooting

Permission Denied (Linux/macOS)

Make the script executable:

chmod +x scrapai

Python Version

Requires Python 3.9 or higher:

python --version  # or python3 --version

Virtual Environment Issues

Delete and recreate:

rm -rf .venv
./scrapai setup

Database Migration Errors

Check DATABASE_URL in .env and ensure database is accessible:

# For SQLite (default)
ls -la scrapai.db

# For PostgreSQL
psql $DATABASE_URL -c "SELECT 1"

Commands

Setup & Verification

setup

Syntax

Options

What It Does

Output

Platform Notes

Linux

Windows

Skip Dependencies

verify

Syntax

What It Checks

Output

Success

Missing Setup

Claude Code Permissions

Allow List

Deny List

Environment Variables

Troubleshooting

Permission Denied (Linux/macOS)

Python Version

Virtual Environment Issues

Database Migration Errors

Next Steps

Spider Management

Crawling

​setup

​Syntax

​Options

​What It Does

​Output

​Platform Notes

​Linux

​Windows

​Skip Dependencies

​verify

​Syntax

​What It Checks

​Output

​Success

​Missing Setup

​Claude Code Permissions

​Allow List

​Deny List

​Environment Variables

​Troubleshooting

​Permission Denied (Linux/macOS)

​Python Version

​Virtual Environment Issues

​Database Migration Errors

​Next Steps

Spider Management

Crawling

setup

Syntax

Options

What It Does

Output

Platform Notes

Linux

Windows

Skip Dependencies

verify

Syntax

What It Checks

Output

Success

Missing Setup

Claude Code Permissions

Allow List

Deny List

Environment Variables

Troubleshooting

Permission Denied (Linux/macOS)

Python Version

Virtual Environment Issues

Database Migration Errors

Next Steps