The Complete Production SOP: How to Deploy and Run Stealth Web Scraper & Harvester in Autonomous AI Agents

---
name: "stealth-scraper"
description: "Harvest dynamic websites using Playwright with anti-detection headers, natural scroll, and structured JSON extraction."
triggers:
  - "scrape website"
  - "extract table data"
  - "crawl ecommerce"
  - "bypass bot check"
---

# Stealth Scraper Procedural SOP
1. Always launch Chromium with `--disable-blink-features=AutomationControlled`.
2. Apply realistic user-agents matching client platform OS.
3. Evaluate page.evaluate("() => delete Object.getPrototypeOf(navigator).webdriver") prior to navigation.
4. Wait for `domcontentloaded` followed by network idle event; avoid arbitrary sleep timers.
5. Extract required content via deterministic CSS/XPath selectors and parse to clean markdown/JSON.

import asyncio
from playwright.async_api import async_playwright
from fake_useragent import UserAgent

async def run_stealth_session(url: str):
    ua = UserAgent(platforms='desktop')
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=['--disable-blink-features=AutomationControlled', '--no-sandbox']
        )
        context = await browser.new_context(
            user_agent=ua.random,
            viewport={'width': 1920, 'height': 1080},
            locale='en-US'
        )
        page = await context.new_page()
        await page.add_init_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
        await page.goto(url, wait_until='networkidle')
        content = await page.content()
        await browser.close()
        return content