---
name: "scrapy-spider-pipelines"
description: "Build enterprise-scale, asynchronous crawling architectures with Scrapy framework."
metadata:
  author: "Scrapy"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "create scrapy spider"
  - "write scrapy pipeline"
  - "configure scrapy autothrottle"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Scrapy Spider Pipelines & Item Loaders

Develops distributed web crawlers using Scrapy: Spider classes, ItemLoaders, middleware pipelines (proxy rotation, User-Agent spoofing), autothrottling, and exporting to JSONL/PostgreSQL.

## When to Invoke
Invoke this skill whenever you need to:
- create scrapy spider
- write scrapy pipeline
- configure scrapy autothrottle

## Execution SOP & Rules
1. Inherit spiders from scrapy.Spider or CrawlSpider with defined rules and LinkExtractors.
2. Extract data cleanly using item loaders: loader.add_css('title', 'h1::text').
3. Process and clean records inside item pipelines in pipelines.py before database persistence.
4. Enable AutoThrottle extension to dynamically adjust crawling speed based on server latency.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.