---
name: "pyarrow-parquet-optimization"
description: "Read, write, and optimize Apache Parquet files with PyArrow and compression tuning."
metadata:
  author: "Apache"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "write optimized parquet pyarrow"
  - "configure zstd compression parquet"
  - "tune row group size parquet"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# PyArrow Snappy & ZSTD Parquet Optimization

Tunes Apache Parquet storage with PyArrow: column chunk sizing, row group budgeting, dictionary encoding, statistics collection, and compression codecs (ZSTD, Snappy).

## When to Invoke
Invoke this skill whenever you need to:
- write optimized parquet pyarrow
- configure zstd compression parquet
- tune row group size parquet

## Execution SOP & Rules
1. Write files using pq.write_table(table, 'out.parquet', compression='ZSTD', compression_level=7).
2. Set row_group_size to balance parallel worker throughput (typically 128MB to 512MB per row group).
3. Enable write_statistics=True to allow query engines to prune row groups using column min/max values.
4. Verify column dictionary encoding is active for low-cardinality string columns.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.