> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Environment Setup

> Configure scrapai CLI environment variables and .env file

## Overview

scrapai CLI uses environment variables stored in a `.env` file in the project root. The file is automatically created during `./scrapai setup` or can be manually created from `.env.example`.

<Note>
  The `.env` file is gitignored by default. Never commit credentials to version control.
</Note>

## Core Environment Variables

### Data Directory

<ParamField path="DATA_DIR" type="string" default="./data">
  Directory where all scraped data, analysis, and artifacts are stored.
</ParamField>

### Database Configuration

<ParamField path="DATABASE_URL" type="string" default="sqlite:///scrapai.db">
  Database connection string. Supports SQLite (default) and PostgreSQL.

  See [Database Configuration](/configuration/database) for details.
</ParamField>

### Logging

<ParamField path="LOG_LEVEL" type="string" default="info">
  Logging verbosity: `debug`, `info`, `warning`, or `error`.
</ParamField>

<ParamField path="LOG_DIR" type="string" default="./logs">
  Directory for log files.
</ParamField>

## Optional Services

### Proxy Configuration

<ParamField path="DATACENTER_PROXY_USERNAME" type="string" required={false}>
  Username for datacenter proxy authentication.

  See [Proxy Configuration](/configuration/proxies) for complete setup.
</ParamField>

<ParamField path="DATACENTER_PROXY_PASSWORD" type="string" required={false}>
  Password for datacenter proxy authentication.
</ParamField>

<ParamField path="DATACENTER_PROXY_HOST" type="string" required={false}>
  Datacenter proxy server hostname.
</ParamField>

<ParamField path="DATACENTER_PROXY_PORT" type="number" required={false}>
  Datacenter proxy server port.
</ParamField>

<ParamField path="RESIDENTIAL_PROXY_USERNAME" type="string" required={false}>
  Username for residential proxy authentication.
</ParamField>

<ParamField path="RESIDENTIAL_PROXY_PASSWORD" type="string" required={false}>
  Password for residential proxy authentication.
</ParamField>

<ParamField path="RESIDENTIAL_PROXY_HOST" type="string" required={false}>
  Residential proxy server hostname.
</ParamField>

<ParamField path="RESIDENTIAL_PROXY_PORT" type="number" required={false}>
  Residential proxy server port.
</ParamField>

### S3 Storage Configuration

<ParamField path="S3_ACCESS_KEY" type="string" required={false}>
  S3-compatible storage access key.

  See [S3 Storage Configuration](/configuration/s3-storage) for complete setup.
</ParamField>

<ParamField path="S3_SECRET_KEY" type="string" required={false}>
  S3-compatible storage secret key.
</ParamField>

<ParamField path="S3_ENDPOINT" type="string" required={false}>
  S3-compatible storage endpoint URL.
</ParamField>

<ParamField path="S3_BUCKET" type="string" required={false}>
  S3 bucket name for storing crawl results.
</ParamField>

## Environment File Example

```bash theme={null}
# Core
DATA_DIR=./data
DATABASE_URL=sqlite:///scrapai.db

# Logging
LOG_LEVEL=info
LOG_DIR=./logs

# Optional: Proxies (see /configuration/proxies)
# DATACENTER_PROXY_USERNAME=
# DATACENTER_PROXY_PASSWORD=
# DATACENTER_PROXY_HOST=
# DATACENTER_PROXY_PORT=

# Optional: S3 Storage (see /configuration/s3-storage)
# S3_ACCESS_KEY=
# S3_SECRET_KEY=
# S3_ENDPOINT=
# S3_BUCKET=
```

## Loading Configuration

Environment variables are automatically loaded from `.env` when you run any scrapai command. Loading order: `.env` file → system environment variables → defaults.

<Tip>
  Override `.env` settings with exported environment variables:

  ```bash theme={null}
  export LOG_LEVEL=debug
  ./scrapai crawl spider_name --project myproject
  ```
</Tip>

## Validation

Verify your configuration:

```bash theme={null}
./scrapai verify
```

## Security Best Practices

<Warning>
  Never commit `.env` to version control. The file contains sensitive credentials and is gitignored by default.
</Warning>

1. Use `.env.example` as a template with placeholder values
2. Restrict permissions: `chmod 600 .env`
3. Rotate credentials regularly
4. Use different credentials per environment (`.env`, `.env.production`, `.env.test`)

## Troubleshooting

**Changes not taking effect:**

* Verify `.env` exists in project root
* Check syntax (no spaces around `=`)
* Restart running processes

**Configuration not found:**

```bash theme={null}
cp .env.example .env
./scrapai setup
```

**Permission denied:**

```bash theme={null}
chmod 600 .env
chmod 755 data/
```

## Related Documentation

* [Database Configuration](/configuration/database)
* [Proxy Configuration](/configuration/proxies)
* [S3 Storage Configuration](/configuration/s3-storage)
* [Installation](/getting-started/installation)