The Complete Production SOP: How to Deploy and Run scrapy-playwright Headless Browser Integration in Autonomous AI Agents

---
name: "scrapy-playwright-headless"
description: "Crawl dynamic JavaScript-heavy single-page applications with scrapy-playwright."
metadata:
  author: "Scrapy Plugins"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "render javascript in scrapy"
  - "use scrapy-playwright"
  - "wait for selector in scrapy"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Execution SOP & Rules
1. Configure DOWNLOAD_HANDLERS in settings.py to route HTTPS through ScrapyPlaywrightDownloadHandler.
2. Specify meta={'playwright': True} on scrapy.Request instances requiring JavaScript execution.
3. Use PageMethod('wait_for_load_state', 'networkidle') or PageMethod('wait_for_selector', selector) for dynamic rendering.
4. Enforce TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor' in settings.py.
5. Cap CONCURRENT_REQUESTS and PLAYWRIGHT_MAX_PAGES_PER_CONTEXT to prevent RAM exhaustion.

# Twisted reactor setup required for scrapy-playwright
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "args": ["--no-sandbox", "--disable-dev-shm-usage"],
}

# Guardrails against host memory exhaustion
CONCURRENT_REQUESTS = 8
PLAYWRIGHT_MAX_CONTEXTS = 4
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 2

import scrapy
from scrapy_playwright.page import PageMethod

class DynamicCatalogSpider(scrapy.Spider):
    name = "dynamic_catalog"
    start_urls = ["https://example.com/spa-products"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url=url,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", "div.product-card"),
                        PageMethod("wait_for_load_state", "networkidle"),
                    ],
                },
                callback=self.parse,
            )

    def parse(self, response):
        for item in response.css("div.product-card"):
            yield {
                "title": item.css("h2.title::text").get(),
                "price": item.css("span.price::text").get(),
            }

#!/usr/bin/env bash
set -euo pipefail

rm -f test_output.json
scrapy crawl dynamic_catalog -s CLOSESPIDER_ITEMCOUNT=1 -o test_output.json

if [ ! -s test_output.json ]; then
  echo "Error: Scrapy-playwright crawl generated empty payload. Target DOM did not hydrate." >&2
  exit 1
fi

echo "Verification Passed: Rendered payload extracted successfully."