The Complete Production SOP: How to Deploy and Run Crawl4AI Asynchronous Deep Extraction Engine in Autonomous AI Agents

---
name: "crawl4ai-deep-llm-extractor"
description: "Execute blazing-fast, open-source asynchronous web crawling optimized for LLMs with Crawl4AI."
metadata:
  author: "unclecode"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "crawl web page crawl4ai"
  - "extract content async crawl4ai"
  - "clean html for llm crawl4ai"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: true
  net_access: true
---

# Crawl4AI Asynchronous Deep Extraction Engine

Crawls and extracts web content with Crawl4AI: asynchronous Playwright execution, heuristic noise filtering, cosine similarity content clustering, and structured JSON extraction via local models.

## Execution SOP & Rules
1. Run crawls asynchronously using `async with AsyncWebCrawler() as crawler: result = await crawler.arun(url)`.
2. Use extraction strategies like `LLMExtractionStrategy` or `CosineStrategy` for structured records.
3. Configure `word_count_threshold` to discard low-content navigation links and footer fragments.
4. Inspect `result.markdown` and `result.extracted_content` for downstream vector ingestion.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel, Field

class PageSchema(BaseModel):
    title: str = Field(description="Primary title of the article or documentation")
    summary: str = Field(description="Three-sentence technical summary")
    code_snippets: list[str] = Field(default_factory=list, description="Extracted code blocks")

async def run_pipeline(target_url: str):
    browser_cfg = BrowserConfig(headless=True, verbose=False)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        word_count_threshold=15,
        remove_overlay_elements=True
    )
    
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(url=target_url, config=run_cfg)
        if result.success:
            print(f"Crawl succeeded. Token-cleaned Markdown length: {len(result.markdown.raw_markdown)}")
            return result.markdown.fit_markdown
        else:
            raise RuntimeError(f"Extraction failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(run_pipeline("https://news.ycombinator.com"))