The Complete Production SOP: How to Deploy and Run Polars Streaming Engine for Larger-Than-RAM Data in Autonomous AI Agents
Production SOP to install and run Polars Streaming Engine for larger-than-RAM data in Codex, Claude Code, & Cursor. Master sink_parquet & out-of-core pipelines.
Technical Overview & Architectural Context
Autonomous coding agents frequently default to naive data manipulation patterns when tasked with transforming large datasets. In Python and Polars environments, LLMs often default to eagerly loading massive files via `pl.read_csv()` or calling `.collect()` prematurely on lazy frames. When processing datasets that scale into hundreds of gigabytes or exceed physical system memory, these unconstrained operations trigger instant out-of-memory (OOM) fatal kills by the OS kernel, crashing the agent runtime container. The `polars-streaming-large-datasets` skill establishes deterministic boundaries for execution pipelines. It forces autonomous agents to construct LazyFrames (`pl.scan_parquet` or `pl.scan_csv`), leverage Polars' Rust-native chunked streaming execution (`lf.collect(streaming=True)`), and push data transformations directly to disk buffers using `lf.sink_parquet()`. By eliminating memory-hungry in-memory transitions and avoiding unconstrained pivots or nested aggregations that break streamability, agents can deterministically process out-of-core data at near C/Rust speeds with minimal RAM footprint.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Mount the skill manifest into your agent's local registry directory so triggers are dynamically indexed.
Agent runtimes such as Claude Code, Cursor, and Codex parse external skills from declarative markdown manifests located within root project configuration directories. By placing `SKILL.md` under `.agents/skills/polars-streaming-large-datasets/` or `.cursor/skills/`, the runtime's intent matching engine evaluates trigger phrases against incoming user prompts and automatically injects streaming execution constraints into system context.
mkdir -p .agents/skills/polars-streaming-large-datasets && cat << 'EOF' > .agents/skills/polars-streaming-large-datasets/SKILL.md
---
name: "polars-streaming-large-datasets"
description: "Process datasets that exceed physical system memory using Polars streaming engine."
metadata:
author: "Polars"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "stream large dataset polars"
- "process out of core data polars"
- "sink parquet polars streaming"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# Polars Streaming Engine for Larger-Than-RAM Data
Configures Polars streaming mode: streaming chunked execution (.collect(streaming=True)), memory-mapped batching, out-of-core sorting, and sink_parquet() direct disk writing.
## Execution SOP & Rules
1. Always scan input data lazily using `pl.scan_parquet()` or `pl.scan_csv()`.
2. Stream transformations directly to disk via `lf.sink_parquet('path.parquet')`.
3. When collecting into memory is unavoidable, invoke `lf.collect(streaming=True)` on filtered subsets.
4. Avoid stream-incompatible operations such as full pivots, multi-column group-by operations with massive cardinalities without spillover settings, or unbounded window joins.
5. Verify that `POLARS_MAX_THREADS` or temporary spill paths are respected during high-concurrency tasks.
EOFimport polars as pl
def run_streaming_pipeline(source_pattern: str, output_path: str) -> None:
# Initialize scan without loading rows into RAM
lazy_df = (
pl.scan_parquet(source_pattern)
.filter(pl.col("status") == "COMPLETED")
.with_columns([
(pl.col("revenue") - pl.col("cost")).alias("margin"),
pl.col("timestamp").dt.truncate("1d").alias("date")
])
.group_by(["date", "category"])
.agg([
pl.col("margin").sum().alias("total_margin"),
pl.col("revenue").sum().alias("total_revenue"),
pl.len().alias("transaction_count")
])
)
# Sink output directly to avoid in-memory dataframe materialization
lazy_df.sink_parquet(
output_path,
compression="zstd",
compression_level=3,
statistics=True
)
if __name__ == "__main__":
run_streaming_pipeline("data/raw_transactions_*.parquet", "data/processed_summary.parquet")Direct your agent to use `explain(streaming=True)` before executing the query. This outputs the query plan directly into stdout, allowing the LLM to inspect whether Polars marked all nodes with `--- STREAMING ---` without consuming runtime memory.
Do not call `.shape` or print the intermediate DataFrame before invoking `sink_parquet()`. Inspecting shapes forces eager evaluation and invalidates lazy streaming, immediately causing RAM spikes.
Configuring Scratch Storage & Thread Environments
Configure runtime memory limits and temp directories to handle out-of-core spillover gracefully.
Out-of-core execution for sorting and grouping in Polars requires designated temporary disk space when memory pools saturate. Setting environment variables ensures the streaming engine does not default to restricted tmpfs virtual mounts that consume volatile memory.
export POLARS_TEMP_DIR="$(pwd)/.tmp_polars"
mkdir -p "$POLARS_TEMP_DIR"
echo "POLARS_TEMP_DIR=$POLARS_TEMP_DIR" >> .env.local
pip install --upgrade "polars>=0.20.0" pyarrowSet `POLARS_VERBOSE=1` during development. The agent can read debug logs from stdout to verify batch sizes and chunk allocations.
Ensure `.tmp_polars` is appended to `.gitignore`. Intermediate sorted spill files can reach dozens of gigabytes and should never be tracked in source control.
Deterministic Prompting & Skill Activation
Trigger agent execution using precise keywords mapped to SKILL.md frontmatter.
Activate the skill deterministically by framing requests with specified action verbs. When the runtime detects phrases like 'stream large dataset polars' or 'sink parquet polars streaming', it activates the streaming rule set over standard Pandas or eager Polars patterns.
Instruct the agent explicitly: 'Verify that the query graph is fully streamable using explain(streaming=True) before executing the pipeline.'
Query Plan Verification & Validation Quality Gate
Validate that the pipeline operates within strict memory thresholds and completes without fallback to eager execution.
Execute an automated self-check script that tests the pipeline on synthetic chunks while enforcing OS-level memory boundaries (e.g., using Python's `resource` module or a container limit).
python -c "import polars as pl; lf = pl.scan_parquet('data/raw_transactions_*.parquet'); plan = lf.explain(streaming=True); assert 'STREAMING' in plan, 'Streaming engine rejected query plan!'"Use `pl.Config.set_streaming_chunk_size(chunk_size)` inside test harnesses to simulate constrained hardware environments during CI runs.
Real-World Scenario: Aggregating 150GB of Web Server Logs on an 8GB RAM Developer Machine
“We have 150GB of compressed web logs in 'data/logs/*.parquet'. We need to calculate unique visitors, total payload bytes, and 4xx/5xx status frequencies per route per hour. Please stream large dataset polars and sink parquet polars streaming directly to 'reports/hourly_metrics.parquet' without running out of RAM.”
Runtime Troubleshooting & Failure Modes
Root Cause: The query plan contains an operator currently unsupported by the streaming engine (e.g., full outer join on unsorted keys, complex window calculations, or non-streamable UDFs).
Split the pipeline: isolate the streamable filtering and pre-aggregation using `sink_parquet()` to an intermediate dataset, then perform the non-streamable operation on the reduced dataset using `.collect()`.Root Cause: Unsorted out-of-core joins or high-cardinality group_by operations exceeded available RAM because scratch disk spilling was disabled or misconfigured.
Configure the streaming engine scratch disk before execution:
import os, polars as pl
os.environ['POLARS_TEMP_DIR'] = '/path/to/fast/ssd/scratch'
# Reduce default chunk size if memory remains tight
pl.Config.set_streaming_chunk_size(50_000)Frequently Asked Questions (Polars Streaming Engine for Larger-Than-RAM Data SOP)
Q:Can I use Polars Streaming Engine for Larger-Than-RAM Data across Codex, Claude Code, and Cursor simultaneously?
Yes. This skill adheres to standard SKILL.md specifications. Simply place the manifest in your project's agent skills directory (`.agents/skills/` or `.cursor/skills/`), and Codex, Claude Code, Cursor, and Windsurf will parse the trigger manifest seamlessly.
Q:What is the key operational difference between lf.collect(streaming=True) and lf.sink_parquet()?
`lf.collect(streaming=True)` streams chunks during intermediate execution but attempts to assemble the final materialized DataFrame inside RAM, which can still trigger OOM errors if the result set is large. `lf.sink_parquet()` streams batches directly to disk, keeping RAM consumption consistently minimal regardless of the input or output size.
Q:Does this skill require escalated root or container permissions?
No. The skill operates completely within userland workspace filesystem boundaries (requiring only read access to source datasets and write access to designated workspace output directories), earning a 98/100 safe security rating.
Ready to equip your autonomous agent with Polars Streaming Engine for Larger-Than-RAM Data?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.