---
name: "apache-spark-pyspark-dataframe-tuning"
description: "Optimize PySpark ETL pipelines: eliminate data skew, minimize shuffle read/write, and tune joins."
metadata:
  author: "Apache Software Foundation"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "tune pyspark partitions"
  - "fix spark data skew salting"
  - "enable spark adaptive query execution"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# PySpark DataFrame Optimization & Partition Tuning

Tunes Apache Spark jobs: repartitioning vs coalescing, broadcast hash joins, handling skewed keys with salting, configuring spark.sql.shuffle.partitions, Adaptive Query Execution (AQE), and cache persistence.

## When to Invoke
Invoke this skill whenever you need to:
- tune pyspark partitions
- fix spark data skew salting
- enable spark adaptive query execution

## Execution SOP & Rules
1. Enable AQE: spark.conf.set('spark.sql.adaptive.enabled', 'true') and spark.sql.adaptive.coalescePartitions.enabled.
2. Use broadcast(small_df) on tables < 100MB to replace SortMergeJoin with BroadcastHashJoin.
3. Salt skewed keys: add salted_key = concat(key, lit('_'), (rand() * 10).cast('int')) before joining.
4. Use df.coalesce(n) to reduce partition count without triggering expensive full network shuffles.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.