The Complete Production SOP: How to Deploy and Run PyArrow Snappy & ZSTD Parquet Optimization in Autonomous AI Agents

---
name: "pyarrow-parquet-optimization"
description: "Read, write, and optimize Apache Parquet files with PyArrow and compression tuning."
metadata:
  author: "Apache"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "write optimized parquet pyarrow"
  - "configure zstd compression parquet"
  - "tune row group size parquet"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# PyArrow Snappy & ZSTD Parquet Optimization

Tunes Apache Parquet storage with PyArrow: column chunk sizing, row group budgeting, dictionary encoding, statistics collection, and compression codecs (ZSTD, Snappy).

## When to Invoke
Invoke this skill whenever you need to:
- write optimized parquet pyarrow
- configure zstd compression parquet
- tune row group size parquet

## Execution SOP & Rules
1. Write files using pq.write_table(table, 'out.parquet', compression='ZSTD', compression_level=7).
2. Set row_group_size to balance parallel worker throughput (typically 128MB to 512MB per row group).
3. Enable write_statistics=True to allow query engines to prune row groups using column min/max values.
4. Verify column dictionary encoding is active for low-cardinality string columns.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.

import sys
import pyarrow as pa
import pyarrow.parquet as pq

def check_environment():
    codecs = ['snappy', 'zstd', 'gzip']
    missing = [c for c in codecs if not pa.Codec.is_available(c)]
    if missing:
        print(f"CRITICAL: Codecs not compiled into pyarrow: {missing}", file=sys.stderr)
        sys.exit(1)
    print("SUCCESS: PyArrow runtime codecs fully operational.")

if __name__ == '__main__':
    check_environment()

import pyarrow as pa
import pyarrow.parquet as pq

def write_optimized_dataset(table: pa.Table, destination_path: str) -> None:
    # SOP Enforcement: ZSTD level 7, 256MB row groups, statistics enabled
    # Calculate estimated row count for ~256MB chunks
    avg_row_size = table.nbytes / max(table.num_rows, 1)
    target_chunk_bytes = 256 * 1024 * 1024
    calculated_row_group_size = max(10000, int(target_chunk_bytes / max(avg_row_size, 1)))

    pq.write_table(
        table,
        destination_path,
        compression='ZSTD',
        compression_level=7,
        row_group_size=calculated_row_group_size,
        write_statistics=True,
        use_dictionary=True,
        data_page_size=1024 * 1024,  # 1MB pages for scan locality
        version='2.6'
    )

import pytest
import pyarrow.parquet as pq

def test_parquet_metadata_compliance():
    meta = pq.read_metadata('out.parquet')
    assert meta.num_row_groups >= 1
    for i in range(meta.num_row_groups):
        rg = meta.row_group(i)
        # Ensure row group uncompressed size is within 64MB - 600MB envelope
        assert rg.total_byte_size < 600 * 1024 * 1024
        for c in range(rg.num_columns):
            col = rg.column(c)
            assert col.compression in ('ZSTD', 'SNAPPY')
            assert col.is_stats_set is True