The Complete Production SOP: How to Deploy and Run Modin Parallel Pandas Acceleration in Autonomous AI Agents

---
name: "modin-parallel-pandas-acceleration"
description: "Accelerate existing Pandas code by 4x across all CPU cores with a 1-line import change."
metadata:
  author: "Modin Project"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "accelerate pandas with modin"
  - "use modin.pandas"
  - "parallelize pd.read_csv"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Modin Parallel Pandas Acceleration

Replaces default single-threaded Pandas with Modin (Ray/Dask engine): distributed DataFrame execution, partition grid matrix, and transparent drop-in API compatibility with zero code rewrites.

## Execution SOP & Rules
1. Replace `import pandas as pd` with `import modin.pandas as pd`.
2. Configure backend engine: `os.environ['MODIN_ENGINE'] = 'ray'` (default) or `'dask'` before importing modin.
3. Verify Ray/Dask initialization parameters (e.g., memory limits and CPU core allocations).
4. Benchmark execution time against standard single-threaded Pandas to verify speedups.
5. Fall back seamlessly to native Pandas for non-implemented experimental methods.

import os
import sys

# Enforce deterministic engine selection prior to import
os.environ["MODIN_ENGINE"] = os.getenv("MODIN_ENGINE", "ray")

try:
    import modin.pandas as pd
    import ray
    if not ray.is_initialized():
        ray.init(ignore_reinit_error=True, include_dashboard=False)
    print(f"[SUCCESS] Modin initialized successfully with backend: {os.environ['MODIN_ENGINE']}")
except ImportError as e:
    print(f"[ERROR] Missing required dependency: {e}", file=sys.stderr)
    sys.exit(1)

import os
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as pd

def process_large_telemetry(csv_path: str) -> pd.DataFrame:
    # Modin reads CSV in parallel partitions across all available physical CPU cores
    df = pd.read_csv(csv_path)
    
    # Filtering and aggregations execute across partitioned blocks
    aggregated = (
        df[df["latency_ms"] > 150]
        .groupby("service_name")
        .agg({"error_count": "sum", "latency_ms": "mean"})
        .reset_index()
    )
    return aggregated

import numpy as np
import pandas as standard_pd
import os
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as modin_pd

def test_modin_pandas_parity(tmp_path):
    test_csv = tmp_path / "benchmark.csv"
    raw_data = {"metric_a": np.random.randn(100000), "metric_b": np.random.randint(0, 100, 100000)}
    standard_pd.DataFrame(raw_data).to_csv(test_csv, index=False)

    # Compare native Pandas output with Modin
    df_standard = standard_pd.read_csv(test_csv).groupby("metric_b").mean()
    df_modin = modin_pd.read_csv(test_csv).groupby("metric_b").mean()

    # Convert modin back to pandas for exact assertion
    standard_pd.testing.assert_frame_equal(df_standard, df_modin._to_pandas())