Skip to main content

​
Overview

scrapai CLI uses environment variables stored in a .env file in the project root. The file is automatically created during ./scrapai setup or can be manually created from .env.example.
The .env file is gitignored by default. Never commit credentials to version control.

​
Core Environment Variables

​
Data Directory

​
DATA_DIR
string
default:"./data"
Directory where all scraped data, analysis, and artifacts are stored.

​
Database Configuration

​
DATABASE_URL
string
default:"sqlite:///scrapai.db"
Database connection string. Supports SQLite (default) and PostgreSQL.See Database Configuration for details.

​
Logging

​
LOG_LEVEL
string
default:"info"
Logging verbosity: debug, info, warning, or error.
​
LOG_DIR
string
default:"./logs"
Directory for log files.

​
Optional Services

​
Proxy Configuration

​
DATACENTER_PROXY_USERNAME
string
Username for datacenter proxy authentication.See Proxy Configuration for complete setup.
​
DATACENTER_PROXY_PASSWORD
string
Password for datacenter proxy authentication.
​
DATACENTER_PROXY_HOST
string
Datacenter proxy server hostname.
​
DATACENTER_PROXY_PORT
number
Datacenter proxy server port.
​
RESIDENTIAL_PROXY_USERNAME
string
Username for residential proxy authentication.
​
RESIDENTIAL_PROXY_PASSWORD
string
Password for residential proxy authentication.
​
RESIDENTIAL_PROXY_HOST
string
Residential proxy server hostname.
​
RESIDENTIAL_PROXY_PORT
number
Residential proxy server port.

​
S3 Storage Configuration

​
S3_ACCESS_KEY
string
S3-compatible storage access key.See S3 Storage Configuration for complete setup.
​
S3_SECRET_KEY
string
S3-compatible storage secret key.
​
S3_ENDPOINT
string
S3-compatible storage endpoint URL.
​
S3_BUCKET
string
S3 bucket name for storing crawl results.

​
Environment File Example

​
Loading Configuration

Environment variables are automatically loaded from .env when you run any scrapai command. Loading order: .env file → system environment variables → defaults.
Override .env settings with exported environment variables:

​
Validation

Verify your configuration:

​
Security Best Practices

Never commit .env to version control. The file contains sensitive credentials and is gitignored by default.
  1. Use .env.example as a template with placeholder values
  2. Restrict permissions: chmod 600 .env
  3. Rotate credentials regularly
  4. Use different credentials per environment (.env, .env.production, .env.test)

​
Troubleshooting

Changes not taking effect:
  • Verify .env exists in project root
  • Check syntax (no spaces around =)
  • Restart running processes
Configuration not found:
Permission denied: