> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider Rules

> URL matching and routing configuration

Rules control which URLs are followed and how they are processed. Each rule defines URL patterns and optional callbacks for extraction.

## SpiderRuleSchema

<ParamField path="allow" type="string[]" default="null">
  Python regex patterns for URLs to allow (non-empty strings)

  **Example:**

  ```json theme={null}
  "allow": ["/news/articles/.*", "/blog/.*"]
  ```
</ParamField>

<ParamField path="deny" type="string[]" default="null">
  Python regex patterns for URLs to deny (takes precedence over allow)

  **Example:**

  ```json theme={null}
  "deny": ["/news/articles/.*#comments", ".*\\?page=.*"]
  ```
</ParamField>

<ParamField path="restrict_xpaths" type="string[]" default="null">
  Only follow links found in these XPath expressions

  **Example:**

  ```json theme={null}
  "restrict_xpaths": ["//div[@class='main-content']//a"]
  ```
</ParamField>

<ParamField path="restrict_css" type="string[]" default="null">
  Only follow links found in these CSS selectors

  **Example:**

  ```json theme={null}
  "restrict_css": ["div.article-list a", "nav.pagination a"]
  ```
</ParamField>

<ParamField path="callback" type="string" default="null">
  Callback function name for processing matched URLs. Must be valid Python identifier and defined in `callbacks` object.

  **Built-in callbacks:**

  * `parse_article` - Extract article content using configured extractors

  **Reserved names:**

  * `parse_article`, `parse_start_url`, `start_requests`, `from_crawler`, `closed`, `parse`

  **Example:**

  ```json theme={null}
  "callback": "parse_product"
  ```

  Use `null` for navigation-only rules (follow links but don't extract).
</ParamField>

<ParamField path="follow" type="boolean" default="true">
  Whether to follow links matching this rule

  **Example:**

  ```json theme={null}
  "follow": false
  ```
</ParamField>

<ParamField path="priority" type="integer" default="0">
  Rule priority (higher = processed first). Range: 0-1000

  **Example:**

  ```json theme={null}
  "priority": 10
  ```
</ParamField>

## Rule Matching

### Allow/Deny Precedence

1. If URL matches any `deny` pattern → **rejected**
2. If URL matches any `allow` pattern → **accepted**
3. If no `allow` patterns defined → **accepted by default**
4. Otherwise → **rejected**

### Restriction Scopes

* `restrict_xpaths` and `restrict_css` limit where links are extracted from
* Links outside these scopes are ignored, even if they match `allow` patterns

## Examples

### News Site

```json theme={null}
{
  "rules": [
    {
      "allow": ["/news/articles/.*"],
      "deny": ["/news/articles/.*#comments"],
      "callback": "parse_article",
      "follow": false
    },
    {
      "allow": ["/news/", "/sport/"],
      "callback": null,
      "follow": true
    }
  ]
}
```

### E-commerce

```json theme={null}
{
  "rules": [
    {
      "allow": ["/product/[^/]+$"],
      "callback": "parse_product",
      "follow": false,
      "priority": 5
    },
    {
      "allow": ["/products", "/category/"],
      "callback": null,
      "follow": true
    }
  ]
}
```

### Forum/Discussion

```json theme={null}
{
  "rules": [
    {
      "allow": ["/item\\?id=\\d+"],
      "deny": ["/vote", "/reply", "/user"],
      "callback": "parse_discussion",
      "follow": false
    }
  ]
}
```

### Restrict Link Sources

```json theme={null}
{
  "rules": [
    {
      "allow": ["/article/.*"],
      "restrict_css": ["div.main-content a", "nav.pagination a"],
      "callback": "parse_article"
    }
  ]
}
```

## Common Patterns

### Exact Path Match

```json theme={null}
"allow": ["/about$", "/contact$"]
```

### Exclude Query Parameters

```json theme={null}
"deny": [".*\\?.*"]
```

### Match Numeric IDs

```json theme={null}
"allow": ["/product/\\d+$", "/job/\\d+$"]
```

### Multiple Domains

```json theme={null}
{
  "allowed_domains": ["example.com", "blog.example.com"],
  "rules": [
    {"allow": ["^https://example\\.com/products/.*"]},
    {"allow": ["^https://blog\\.example\\.com/posts/.*"]}
  ]
}
```

### Pagination

```json theme={null}
{
  "allow": ["/page/\\d+$"],
  "callback": null,
  "follow": true
}
```

## Rule Order

Rules are processed in array order. Use `priority` to control processing order within Scrapy.

## Validation Errors

### Undefined Callback

```
Rule 0 references undefined callback: 'parse_product'. 
Defined callbacks: parse_article
```

**Fix:** Add callback to `callbacks` object or use `parse_article`

### Invalid Callback Name

```
Invalid callback name: 'parse-product'. 
Must be a valid Python identifier.
```

**Fix:** Use underscores instead of hyphens: `parse_product`

### Empty Patterns

```
Patterns must be non-empty strings
```

**Fix:** Remove empty strings from `allow`/`deny` arrays

## Related

* [Spider Schema](/api/spider-schema) - Complete configuration reference
* [Callbacks](/api/callbacks) - Custom extraction configuration
* [Settings](/api/settings) - Spider behavior settings