---
name: "openlineage-metadata-data-lineage"
description: "Collect end-to-end dataset lineage and job run metadata across Airflow, Spark, and dbt."
metadata:
  author: "OpenLineage"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure openlineage airflow"
  - "collect spark data lineage"
  - "stream lineage to marquez"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# OpenLineage Operational Data Lineage & Governance

Integrates OpenLineage: configuring Airflow OpenLineage provider, Spark OpenLineage agents, extracting dataset inputs/outputs and column-level facets, and streaming lineage events to Marquez or Atlan.

## When to Invoke
Invoke this skill whenever you need to:
- configure openlineage airflow
- collect spark data lineage
- stream lineage to marquez

## Execution SOP & Rules
1. Install openlineage-airflow and set OPENLINEAGE_URL and OPENLINEAGE_NAMESPACE environment variables.
2. For Spark: submit jobs with --packages io.openlineage:openlineage-spark:... and extraListeners configuration.
3. Inspect generated RunEvent JSON payloads containing Job, Run, and Inputs/Outputs dataset facets.
4. Query Marquez REST API or UI to audit data transformations and assess blast radius of pipeline failures.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.