> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Commands

> View and export scraped items from database

Data commands allow you to inspect scraped items from test crawls and export them to various formats (CSV, JSON, JSONL, Parquet).

## show

View scraped items from the database.

### Syntax

```bash theme={null}
./scrapai show <spider> --project <name> [options]
```

### Arguments

<ParamField path="spider" type="string" required>
  Spider name.
</ParamField>

### Options

<ParamField path="--project" type="string" required>
  Project name.
</ParamField>

<ParamField path="--limit, -l" type="integer" default="5">
  Number of items to display.
</ParamField>

<ParamField path="--url" type="string">
  Filter by URL pattern (case-insensitive substring match).
</ParamField>

<ParamField path="--title" type="string">
  Search in titles only (case-insensitive).
</ParamField>

<ParamField path="--text, -t" type="string">
  Search in both title and content (case-insensitive).
</ParamField>

### Examples

```bash theme={null}
# Show last 5 items
./scrapai show bbc_co_uk --project news

# Show last 10 items
./scrapai show bbc_co_uk --project news --limit 10

# Filter by URL pattern
./scrapai show bbc_co_uk --project news --url "/technology/"

# Search titles
./scrapai show bbc_co_uk --project news --title "climate"

# Search title and content
./scrapai show bbc_co_uk --project news --text "election"
```

### Output

#### Article Items (Generic Extractors)

```bash theme={null}
$ ./scrapai show bbc_co_uk --project news --limit 3
📰 Showing 3 articles from 'bbc_co_uk':

🔸 [1] UK economy grows 0.4% in February
   📅 Published: 2026-02-28 | Scraped: 2026-02-28 15:30
   🔗 https://bbc.co.uk/news/business-123456
   ✍️  By Economics Reporter
   📝 The UK economy grew by 0.4% in February, official figures show, 
        beating economists' expectations of 0.2% growth...

🔸 [2] NASA announces Mars mission timeline
   📅 Published: 2026-02-28 | Scraped: 2026-02-28 15:32
   🔗 https://bbc.co.uk/news/science-789012
   📝 NASA has unveiled an ambitious timeline for its first crewed mission 
        to Mars, targeting a 2035 launch date...

🔸 [3] New AI regulations announced
   📅 Published: 2026-02-27 | Scraped: 2026-02-28 15:35
   🔗 https://bbc.co.uk/news/technology-345678
   ✍️  By Tech Correspondent
   📝 The government has announced new regulations for artificial intelligence 
        systems, focusing on transparency and accountability...
```

#### Callback Items (Custom Extractors)

```bash theme={null}
$ ./scrapai show ecommerce_spider --project shop --limit 2
📰 Showing 2 articles from 'ecommerce_spider':

🔸 [1] parse_product item
   📅 Scraped: 2026-02-28 16:10
   🔗 https://shop.example.com/products/widget-pro-3000
   • title: Widget Pro 3000
   • price: $299.99
   • rating: 4.5
   • reviews_count: 1,247
   • availability: In Stock
```

## export

Export scraped items to file formats.

### Syntax

```bash theme={null}
./scrapai export <spider> --project <name> --format <fmt> [options]
```

### Arguments

<ParamField path="spider" type="string" required>
  Spider name.
</ParamField>

### Options

<ParamField path="--project" type="string" required>
  Project name.
</ParamField>

<ParamField path="--format, -f" type="choice" required>
  Export format: `csv`, `json`, `jsonl`, `parquet`.
</ParamField>

<ParamField path="--output, -o" type="string">
  Custom output file path. If not specified, uses timestamped filename in `data/<project>/<spider>/exports/`.
</ParamField>

<ParamField path="--limit, -l" type="integer">
  Limit number of items to export.
</ParamField>

<ParamField path="--url" type="string">
  Filter by URL pattern.
</ParamField>

<ParamField path="--title" type="string">
  Filter by title.
</ParamField>

<ParamField path="--text, -t" type="string">
  Filter by title or content.
</ParamField>

### Examples

```bash theme={null}
# Export to CSV (default location)
./scrapai export bbc_co_uk --project news --format csv

# Export to JSON with custom path
./scrapai export bbc_co_uk --project news --format json --output my_data.json

# Export to Parquet (requires pandas)
./scrapai export bbc_co_uk --project news --format parquet

# Export filtered items
./scrapai export bbc_co_uk --project news --format jsonl --title "climate"

# Export limited items
./scrapai export bbc_co_uk --project news --format csv --limit 100
```

### Output

```bash theme={null}
$ ./scrapai export bbc_co_uk --project news --format csv
✅ Exported 247 articles to CSV: data/news/bbc_co_uk/exports/export_28022026_153042.csv
```

### Export Formats

#### CSV

```csv theme={null}
id,url,title,content,author,published_date,scraped_at
1,https://bbc.co.uk/news/business-123456,"UK economy grows 0.4%","The UK economy...","Economics Reporter",2026-02-28,2026-02-28T15:30:42
2,https://bbc.co.uk/news/science-789012,"NASA announces Mars mission","NASA has unveiled...","",2026-02-28,2026-02-28T15:32:15
```

#### JSON

Pretty-printed JSON array:

```json theme={null}
[
  {
    "id": 1,
    "url": "https://bbc.co.uk/news/business-123456",
    "title": "UK economy grows 0.4%",
    "content": "The UK economy...",
    "author": "Economics Reporter",
    "published_date": "2026-02-28",
    "scraped_at": "2026-02-28T15:30:42"
  },
  {
    "id": 2,
    "url": "https://bbc.co.uk/news/science-789012",
    "title": "NASA announces Mars mission",
    "content": "NASA has unveiled...",
    "author": null,
    "published_date": "2026-02-28",
    "scraped_at": "2026-02-28T15:32:15"
  }
]
```

#### JSONL (JSON Lines)

```jsonl theme={null}
{"id": 1, "url": "https://bbc.co.uk/news/business-123456", "title": "UK economy grows 0.4%", "scraped_at": "2026-02-28T15:30:42"}
{"id": 2, "url": "https://bbc.co.uk/news/science-789012", "title": "NASA announces Mars mission", "scraped_at": "2026-02-28T15:32:15"}
```

#### Parquet

Requires `pandas` and `pyarrow`:

```bash theme={null}
.venv/bin/pip install pandas pyarrow
./scrapai export bbc_co_uk --project news --format parquet
```

### Default Export Location

```
data/<project>/<spider>/exports/export_<timestamp>.<format>
```

## Database Storage

### `scraped_items` Table

Items are stored in this table during test crawls:

```sql theme={null}
CREATE TABLE scraped_items (
    id INTEGER PRIMARY KEY,
    spider_id INTEGER NOT NULL,
    url VARCHAR(2048) NOT NULL,
    title TEXT,
    content TEXT,
    author VARCHAR(255),
    published_date TIMESTAMP,
    scraped_at TIMESTAMP NOT NULL,
    metadata_json JSON,
    FOREIGN KEY (spider_id) REFERENCES spiders(id) ON DELETE CASCADE
);
```

### Standard vs. Custom Fields

**Standard fields** (from newspaper/trafilatura extractors):

* Stored in dedicated columns: `title`, `content`, `author`, `published_date`

**Custom fields** (from callback extractors):

* Stored in `metadata_json` column as JSON
* Includes `_callback` key to identify which callback was used

## Data Retention

**Test Crawls**: Data stored in database until spider is deleted (cascading delete).

**Production Crawls**: Data exported to JSONL files in `data/<project>/<spider>/crawls/` (not stored in database).

## Troubleshooting

**Spider Not Found**: Verify project and spider name with `./scrapai spiders list --project <name>`

**No Items Found**: Run a test crawl first - production crawls save to JSONL files, not database

**Parquet Export Error**: Install dependencies: `.venv/bin/pip install pandas pyarrow`

## Next Steps

<CardGroup cols={2}>
  <Card title="Database Commands" icon="server" href="/cli/database">
    Advanced queries and database management
  </Card>

  <Card title="Inspection" icon="magnifying-glass" href="/cli/inspect">
    Analyze websites before scraping
  </Card>
</CardGroup>