> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Projects Commands

> List and manage project organization

Projects provide logical grouping for spiders and queue items. Each project is a namespace that keeps related spiders organized.

## projects list

List all projects in the database.

### Syntax

```bash theme={null}
./scrapai projects list
```

### Output

```bash theme={null}
$ ./scrapai projects list
📁 Available Projects:
  • news
    Spiders: 8, Queue items: 23
  • ecommerce
    Spiders: 12, Queue items: 47
  • research
    Spiders: 5, Queue items: 0
  • archive
    Spiders: 3, Queue items: 2
```

### Information Shown

* **Project name**: Identifier used in all commands
* **Spider count**: Number of spiders in this project
* **Queue items**: Number of items in queue for this project

### Empty Database

```bash theme={null}
$ ./scrapai projects list
No projects found.
```

## Project Organization

Projects are **implicit** - they don't need to be created explicitly. When you import a spider or add a queue item with `--project <name>`, the project is automatically associated.

```bash theme={null}
# Import spider creates project "news"
./scrapai spiders import bbc_spider.json --project news

# Add queue item creates project "research"
./scrapai queue add https://example.com --project research
```

If `--project` is not specified, commands use `default`.

<Tip>
  Always use explicit `--project` names for clarity. Avoid relying on the default.
</Tip>

## Project Naming

Project names should be:

* Descriptive: `news`, `ecommerce`, `research`
* Lowercase: `news` not `News`
* No spaces: `tech_blogs` not `tech blogs`
* Consistent: Choose a naming scheme and stick to it

## Use Cases

### Multi-Domain Projects

Group related spiders:

```bash theme={null}
# News aggregation project
./scrapai spiders import bbc.json --project news
./scrapai spiders import cnn.json --project news
./scrapai spiders import reuters.json --project news

# E-commerce monitoring project
./scrapai spiders import amazon.json --project ecommerce
./scrapai spiders import ebay.json --project ecommerce
```

### Team Organization

Separate projects by team or purpose:

```bash theme={null}
# Marketing team
./scrapai spiders import competitors.json --project marketing

# Research team
./scrapai spiders import papers.json --project research

# Sales team
./scrapai spiders import leads.json --project sales
```

### Development vs Production

Separate test and production spiders:

```bash theme={null}
# Development/testing
./scrapai spiders import test_spider.json --project dev
./scrapai crawl test_spider --project dev --limit 5

# Production
./scrapai spiders import prod_spider.json --project prod
./scrapai crawl prod_spider --project prod
```

## Project-Level Operations

### List Spiders by Project

```bash theme={null}
./scrapai spiders list --project news
```

### Run All Spiders in Project

```bash theme={null}
./scrapai crawl-all --project news
```

### Queue Management by Project

```bash theme={null}
# Add to project queue
./scrapai queue add https://example.com --project news

# List project queue
./scrapai queue list --project news

# Process project queue
./scrapai queue next --project news
```

### Export Project Data

```bash theme={null}
# Export all spiders in project
for spider in $(./scrapai spiders list --project news | grep '•' | awk '{print $2}'); do
  ./scrapai export $spider --project news --format csv
done
```

## Data Organization

Files are organized by project:

```
data/
├── news/
│   ├── bbc_co_uk/
│   │   ├── crawls/
│   │   ├── exports/
│   │   └── checkpoint/
│   ├── cnn_com/
│   └── reuters_com/
├── ecommerce/
│   ├── amazon_spider/
│   └── ebay_spider/
└── research/
    └── papers_spider/
```

## Database Schema

Projects are stored as string fields:

### `spiders` Table

```sql theme={null}
CREATE TABLE spiders (
    id INTEGER PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    project VARCHAR(255),  -- Project name
    ...
);
```

### `crawl_queue` Table

```sql theme={null}
CREATE TABLE crawl_queue (
    id INTEGER PRIMARY KEY,
    project_name VARCHAR(255) NOT NULL,  -- Project name
    website_url VARCHAR(2048) NOT NULL,
    ...
);
```

### Project Uniqueness

* **Spiders**: Unique by `(name, project)` - same spider name can exist in different projects
* **Queue**: Unique by `(project_name, website_url)` - same URL can be in different projects

## Renaming Projects

Projects can be renamed via direct database updates:

```bash theme={null}
# Rename project "old_name" to "new_name"
./scrapai db query "UPDATE spiders SET project='new_name' WHERE project='old_name'" --yes
./scrapai db query "UPDATE crawl_queue SET project_name='new_name' WHERE project_name='old_name'" --yes
```

<Warning>
  Manually renaming projects does not rename the data directories. You'll need to rename those separately:

  ```bash theme={null}
  mv data/old_name data/new_name
  ```
</Warning>

## Deleting Projects

Delete all spiders in a project:

```bash theme={null}
# List spiders first
./scrapai spiders list --project old_project

# Delete each spider
./scrapai spiders delete spider1 --project old_project --force
./scrapai spiders delete spider2 --project old_project --force
```

Clean up queue:

```bash theme={null}
./scrapai queue cleanup --project old_project --all --force
```

Remove data directory:

```bash theme={null}
rm -rf data/old_project
```

## Parallel Processing by Project

Run multiple projects in parallel:

```bash theme={null}
# Terminal 1: News project
while true; do
  ./scrapai queue next --project news && # process
done

# Terminal 2: E-commerce project
while true; do
  ./scrapai queue next --project ecommerce && # process
done

# Terminal 3: Research project
while true; do
  ./scrapai queue next --project research && # process
done
```

## Project Statistics

Get detailed stats per project:

```bash theme={null}
# Spider count
./scrapai db query "SELECT project, COUNT(*) FROM spiders GROUP BY project"

# Item count per project
./scrapai db query "
SELECT s.project, COUNT(si.id) as items
FROM spiders s
LEFT JOIN scraped_items si ON s.id = si.spider_id
GROUP BY s.project
ORDER BY items DESC
"

# Queue status by project
./scrapai db query "
SELECT project_name, status, COUNT(*) 
FROM crawl_queue 
GROUP BY project_name, status
"
```

## Best Practices

* **Use descriptive names**: `tech_news_monitoring` not `proj1`
* **Separate dev and prod**: Use `dev_` and `prod_` prefixes
* **Document project purpose**: Maintain a `projects.md` file describing each project
* **Consistent naming**: Choose a convention (simple names, prefixed names, or dated names) and stick to it

## Migrating Between Projects

```bash theme={null}
# Update project in database
./scrapai db query "UPDATE spiders SET project='new_project' WHERE project='old_project'" --yes

# Move data directory
mv data/old_project data/new_project
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Spider Management" icon="spider" href="/cli/spiders">
    Import spiders into projects
  </Card>

  <Card title="Queue Management" icon="list" href="/cli/queue">
    Add URLs to project queues
  </Card>
</CardGroup>