Overview
scrapai CLI uses environment variables stored in a.env file in the project root. The file is automatically created during ./scrapai setup or can be manually created from .env.example.
The
.env file is gitignored by default. Never commit credentials to version control.Core Environment Variables
Data Directory
string
default:"./data"
Directory where all scraped data, analysis, and artifacts are stored.
Database Configuration
string
default:"sqlite:///scrapai.db"
Database connection string. Supports SQLite (default) and PostgreSQL.See Database Configuration for details.
Logging
string
default:"info"
Logging verbosity:
debug, info, warning, or error.string
default:"./logs"
Directory for log files.
Optional Services
Proxy Configuration
string
Username for datacenter proxy authentication.See Proxy Configuration for complete setup.
string
Password for datacenter proxy authentication.
string
Datacenter proxy server hostname.
number
Datacenter proxy server port.
string
Username for residential proxy authentication.
string
Password for residential proxy authentication.
string
Residential proxy server hostname.
number
Residential proxy server port.
S3 Storage Configuration
string
S3-compatible storage access key.See S3 Storage Configuration for complete setup.
string
S3-compatible storage secret key.
string
S3-compatible storage endpoint URL.
string
S3 bucket name for storing crawl results.
Environment File Example
Loading Configuration
Environment variables are automatically loaded from.env when you run any scrapai command. Loading order: .env file → system environment variables → defaults.
Validation
Verify your configuration:Security Best Practices
- Use
.env.exampleas a template with placeholder values - Restrict permissions:
chmod 600 .env - Rotate credentials regularly
- Use different credentials per environment (
.env,.env.production,.env.test)
Troubleshooting
Changes not taking effect:- Verify
.envexists in project root - Check syntax (no spaces around
=) - Restart running processes