---
name: "scrapy-playwright-headless"
description: "Crawl dynamic JavaScript-heavy single-page applications with scrapy-playwright."
metadata:
  author: "Scrapy Plugins"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "render javascript in scrapy"
  - "use scrapy-playwright"
  - "wait for selector in scrapy"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# scrapy-playwright Headless Browser Integration

Bridges Scrapy asynchronous pipelines with Playwright: rendering client-rendered pages, waiting for network idle, executing custom page evaluate scripts, and intercepting API calls.

## When to Invoke
Invoke this skill whenever you need to:
- render javascript in scrapy
- use scrapy-playwright
- wait for selector in scrapy

## Execution SOP & Rules
1. Configure DOWNLOAD_HANDLERS in settings.py to route HTTPS through ScrapyPlaywrightDownloadHandler.
2. Specify meta={'playwright': True} on scrapy.Request instances that require JavaScript execution.
3. Use PageMethod('wait_for_load_state', 'networkidle') to ensure dynamic content finishes rendering.
4. Limit concurrency when using Playwright downloaders to manage server RAM consumption.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.