---
name: "spark-structured-streaming-watermarks"
description: "Ingest, transform, and aggregate continuous event streams with Spark Structured Streaming."
metadata:
  author: "Apache Software Foundation"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "spark structured streaming kafka"
  - "configure spark streaming watermark"
  - "write stream to delta spark"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Spark Structured Streaming Watermarks & Sinks

Builds streaming jobs with Apache Spark: consuming from Kafka, defining event-time watermarks (withWatermark), sliding window aggregations, output modes (append, update, complete), and checkpointing to S3.

## When to Invoke
Invoke this skill whenever you need to:
- spark structured streaming kafka
- configure spark streaming watermark
- write stream to delta spark

## Execution SOP & Rules
1. Read stream: df = spark.readStream.format('kafka').option('kafka.bootstrap.servers', '...').load().
2. Define watermark: df.withWatermark('timestamp', '10 minutes').groupBy(window('timestamp', '5 minutes'), 'page_id').count().
3. Write stream with checkpoint: query = df.writeStream.format('delta').outputMode('append').option('checkpointLocation', '/path').start().
4. Monitor query progress: query.recentProgress and handle graceful shutdown on SIGTERM.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.