The Complete Production SOP: How to Deploy and Run Readability Python Primary Content Extractor in Autonomous AI Agents
Step-by-step SOP to install and execute the readability-lxml-content-cleaner skill in Codex, Claude Code, and Cursor. Clean HTML and isolate article bodies.
Technical Overview & Architectural Context
Autonomous coding agents frequently fail when ingesting raw web pages for Retrieval-Augmented Generation (RAG) pipelines, data scraping tasks, and document summarization. When tasked with extracting article content, agents commonly default to naive regex parsers or unbounded BeautifulSoup tag removals that indiscriminately strip semantic headers, preserve bloated navigation bars, retain tracking pixels, or completely wipe dynamic layout wrappers. This consumes thousands of redundant context tokens and introduces noise into downstream LLM embeddings. The 'readability-lxml-content-cleaner' skill solves this by enforcing Mozilla Readability's battle-tested DOM text-to-tag heuristics through python-readability and lxml. Instead of allowing the agent to guess structural selectors, the SKILL.md injects deterministic procedural rules: initializing the Document parser, resolving candidate nodes by density scoring, isolating doc.title() and doc.summary(), and validating minimum length thresholds. If content density falls below acceptable limits, the SOP instructs the agent to execute clean fallback extraction rather than returning empty payloads.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local agent skill manifest directory and inject verified procedural rules into your workspace.
Autonomous agent runtimes inspect the `.agents/skills` or `.claude/skills` directories on boot to register contextual triggers and operational boundaries. By placing `readability-lxml-content-cleaner` directly in your workspace repository, you ensure that Codex, Claude Code, Cursor, and Windsurf intercept user queries matching 'extract article with readability' and restrict execution to local deterministic parsers without making unapproved outbound network calls.
mkdir -p .agents/skills/readability-lxml-content-cleaner && cat << 'EOF' > .agents/skills/readability-lxml-content-cleaner/SKILL.md
---
name: "readability-lxml-content-cleaner"
description: "Isolate primary readable content from complex web layouts using Mozilla Readability port."
metadata:
author: "Buriy"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "extract article with readability"
- "isolate main content html"
- "strip web page sidebars"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# Readability Python Primary Content Extractor
Leverages python-readability (lxml): identifying the primary content container based on text-to-tag density heuristics, stripping sidebars, and extracting clean article HTML titles and summaries.
## When to Invoke
Invoke this skill whenever you need to:
- extract article with readability
- isolate main content html
- strip web page sidebars
## Execution SOP & Rules
1. Initialize document with doc = Document(raw_html).
2. Retrieve clean article HTML with doc.summary() and title with doc.title().
3. Strip remaining styling attributes and convert to Markdown for LLM summarization.
4. Handle pages with low text density by falling back to full DOM extraction.
## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.
EOF---
name: "readability-lxml-content-cleaner"
description: "Isolate primary readable content from complex web layouts using Mozilla Readability port."
metadata:
author: "Buriy"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "extract article with readability"
- "isolate main content html"
- "strip web page sidebars"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---Keep the SKILL.md under 150 lines. Restricting triggers and rules to clear procedural constraints prevents agent hallucination and conserves context window capacity during deep refactoring loops.
Ensure permissions.net_access remains false in the frontmatter. Readability is an offline parser designed to operate strictly on raw HTML files or pre-fetched strings within workspace bounds.
Environment Dependencies & Parser Engine Installation
Install the required C-bindings and python-readability packages within an isolated virtual environment.
The underlying engine relies on lxml, which binds directly to system-level libxml2 libraries. If the runtime sandbox lacks these compiled binaries, lxml imports will fail silently or abort compilation. Run the following installation commands within your virtual environment to ensure proper dependency mounting.
python3 -m venv .venv && source .venv/bin/activate && pip install --upgrade pip && pip install readability-lxml lxml beautifulsoup4 markdownifyfrom readability import Document
from bs4 import BeautifulSoup
import markdownify
def clean_webpage(raw_html: str) -> dict:
doc = Document(raw_html)
clean_html = doc.summary()
clean_title = doc.title()
# Fallback heuristic if text density is insufficient
soup = BeautifulSoup(clean_html, 'html.parser')
text_content = soup.get_text(strip=True)
if len(text_content) < 100:
fallback_soup = BeautifulSoup(raw_html, 'html.parser')
for tag in fallback_soup(['script', 'style', 'nav', 'footer', 'aside']):
tag.decompose()
clean_html = str(fallback_soup.body or fallback_soup)
text_content = fallback_soup.get_text(strip=True)
markdown_output = markdownify.markdownify(clean_html, heading_style="ATX")
return {
"title": clean_title,
"html": clean_html,
"markdown": markdown_output,
"char_count": len(text_content)
}Pre-compile lxml wheels on Debian/Alpine containers by installing python3-dev and gcc beforehand to prevent build timeouts in constrained agent sandboxes.
Do not install the generic 'readability' package from PyPI; it is deprecated and unmaintained. Always explicitly install 'readability-lxml'.
Intent Matching & Trigger Invocation
Execute user prompts that reliably activate the skill across Codex, Claude Code, Cursor, and Windsurf.
Supported AI agent runtimes match user queries against the triggers declared in SKILL.md. Phrasing queries with exact trigger phrases guarantees that the runtime selects the readability-lxml-content-cleaner policy rather than attempting arbitrary ad-hoc DOM manipulation.
# Execute via Claude Code CLI
claude "isolate main content html from data/raw_articles/sample_blog.html and strip web page sidebars"
# Or within Cursor Composer / Codex Desktop
# Prompt: 'extract article with readability for all stored landing pages in ./scrapes'Prefix queries with explicit workspace paths and verbs that directly correspond to the frontmatter triggers to achieve near-100% deterministic skill selection.
If an agent tries to invoke curl or requests within the script, remind it that SKILL.md defines net_access: false and it must only ingest local files.
Verification, Linting & Output Quality Gate
Run deterministic syntax, typing, and extraction quality gates to ensure clean deliverables.
Before completing a task, the agent must verify its extraction pipeline against edge-case markup (e.g., empty tags, malformed UTF-8, script-heavy single-page app snapshots) using automated test assertions.
python -m pytest tests/test_extractor.py -v && ruff check extractor_service.pyimport pytest
from extractor_service import clean_webpage
def test_clean_webpage_strips_boilerplate():
dirty_html = """
<!DOCTYPE html>
<html>
<head><title>Test Article Page</title></head>
<body>
<header><nav>Home | About | Contact</nav></header>
<aside class="sidebar">Advertisements and Related Links</aside>
<article id="main-article">
<h1>Production Ready AI Agent Workflows</h1>
<p>Deterministic procedural skills improve autonomous execution quality significantly.</p>
<p>Readability isolates this node while discarding surrounding noise efficiently.</p>
</article>
<footer>Copyright 2026 Sandbox Inc.</footer>
</body>
</html>
"""
result = clean_webpage(dirty_html)
assert result["title"] == "Test Article Page"
assert "Advertisements and Related Links" not in result["markdown"]
assert "Deterministic procedural skills improve autonomous execution" in result["markdown"]
assert result["char_count"] > 50Ensure tests include checks verifying that navigational landmarks and advertising boilerplate are stripped from the markdown output.
Relying strictly on doc.summary() without fallbacks will return empty DIVs on pages heavily reliant on microdata tables or non-standard semantic blocks.
Real-World Scenario: Batch Processing Raw Documentation Pages for Agent RAG Ingestion
“We have 150 scraped HTML files in ./data/raw_docs. Please isolate main content html, strip web page sidebars, and extract article with readability into clean markdown files under ./data/clean_docs. Ensure each file retains its title as an H1 heading.”
Runtime Troubleshooting & Failure Modes
Root Cause: The agent passed an empty HTML file or an invalid binary string to the Document object without performing a preliminary content length check.
if not raw_html or not raw_html.strip():
raise ValueError('Input HTML string is empty or contains only whitespace.')
doc = Document(raw_html)Root Cause: The incorrect 'readability' library was installed instead of the required 'readability-lxml' fork.
pip uninstall -y readability && pip install readability-lxmlFrequently Asked Questions (Readability Python Primary Content Extractor SOP)
Q:Can I use Readability Python Primary Content Extractor across Codex, Claude Code, and Cursor simultaneously?
Yes. This skill is authored strictly according to the SKILL.md specification. All modern agents capable of reading workspace agent rules (.agents/skills/ or .claude/skills/) can mount and execute it deterministically across different platforms.
Q:How does Readability Python Primary Content Extractor prevent context window overflow during long agent sessions?
Raw HTML files routinely consume 50,000 to 100,000 tokens due to inline SVG, CSS styles, JavaScript bundles, and deep DOM layouts. By isolating only the primary content container and converting it to clean Markdown, this skill reduces token usage by 85% to 95% before data enters the agent context.
Q:Does this skill require escalated root or container permissions?
No. The skill has an audited security score of 98/100 and strictly enforces fs_write within the active workspace. It requires no shell execution permissions and blocks outbound network access, relying entirely on local parser execution.
Ready to equip your autonomous agent with Readability Python Primary Content Extractor?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.