The Complete Production SOP: How to Deploy and Run Apache Iceberg Hidden Partitioning & Metadata Evolution in Autonomous AI Agents

---
name: "apache-iceberg-hidden-partitioning"
description: "Manage high-performance open table formats with hidden partitioning and schema evolution."
metadata:
  author: "Apache Software Foundation"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "create iceberg table"
  - "configure iceberg hidden partitioning"
  - "evolve iceberg schema"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Execution SOP & Rules
1. Enforce hidden partitioning: Use transforms `days(ts)`, `hours(ts)`, `bucket(N, id)`, or `truncate(L, str)` directly in the `PARTITIONED BY` clause.
2. Prevent synthetic partition columns: Do NOT add derived string columns like `p_date` or `dt`.
3. Execute metadata-only schema evolution: Use `ALTER TABLE ... ADD COLUMN` or `ALTER TABLE ... ALTER COLUMN` without data rewrites.
4. Evolve partitioning via metadata: Alter partition specs using `ALTER TABLE ... ADD PARTITION FIELD` or `DROP PARTITION FIELD` without rewriting historical data.
5. Verify snapshot isolation: Query metadata log files and `.metadata.json` pointers to confirm atomic commit integrity.

import os
from pyiceberg.catalog import load_catalog

# Deterministic catalog loader using environment variables
catalog_name = os.getenv("ICEBERG_CATALOG_NAME", "default")
try:
    catalog = load_catalog(catalog_name)
    print(f"Successfully connected to Iceberg catalog: {catalog_name}")
except Exception as err:
    print(f"Catalog initialization check: {err}")

-- Verified Iceberg DDL generated under skill boundaries
CREATE TABLE iceberg_catalog.telemetry.events (
    event_id UUID,
    user_id BIGINT,
    event_time TIMESTAMP,
    payload STRING
)
USING iceberg
PARTITIONED BY (
    days(event_time),
    bucket(16, user_id)
)
TBLPROPERTIES (
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'zstd',
    'history.expire.max-snapshot-age-ms' = '604800000'
);