Verified SOP & TutorialAudited Score 98/100

The Complete Production SOP: How to Deploy and Run Debezium CDC Database Replication to Kafka in Autonomous AI Agents

Step-by-step SOP to install and run Debezium CDC replication in Codex, Claude Code, and Cursor. Verified Kafka Connect configs, pgoutput setup, and troubleshooting.

View 1-Click Spec

Target Runtimes:codexclaude-codecursorwindsurf

5 min read • Updated for 2026

Technical Overview & Architectural Context

Autonomous coding agents frequently stumble when provisioning Change Data Capture (CDC) pipelines. Without strict deterministic guardrails, models confuse legacy PostgreSQL test decoders with modern pgoutput engines, omit mandatory database parameters like wal_level=logical or max_replication_slots, generate invalid Kafka Connect JSON manifests, and fail to account for tombstone deletion events and schema evolution registries. The 'debezium-cdc-change-data-capture' skill installs rigorous procedural standard operating procedures directly into agent memory across Codex, Claude Code, Cursor, and Windsurf. It forces agents to follow an authoritative validation pipeline: verifying PostgreSQL replication privileges, formulating robust Kafka Connect payloads with filtered publication autocreation, and running health checks against the Kafka Connect REST API endpoint (/connectors/{name}/status). By constraining agent actions to explicit workspace configurations and validated REST payloads, this skill completely eliminates hallucinated connector configurations, unmonitored replication slot bloat, and broken downstream Kafka stream consumers.

Prerequisites & Environment Requirements

Codex Desktop, Claude Code CLI, Cursor, or Windsurf installed with active workspace

Docker & Docker Compose installed (running Kafka, Kafka Connect, and PostgreSQL 14+)

curl, jq, and PostgreSQL client utilities (psql) installed locally

Verified Search Intents & Long-Tail Queries Solved

SEO Audit Score: 98/100

This operational tutorial is engineered to provide deterministic solutions for these developer search queries:

#install debezium-cdc-change-data-capture in cursor ide#claude code debezium-cdc-change-data-capture rules#codex desktop debezium-cdc-change-data-capture setup tutorial#configure debezium postgres connector kafka connect api#debezium postgresql logical replication pgoutput slot#implement outbox pattern cdc kafka debezium#debezium-cdc-change-data-capture sandbox permissions and security score#debezium-cdc-change-data-capture prompt examples and triggers

Step-by-Step Execution Playbook

Local Environment Setup & SKILL.md Mounting

Mount the Debezium CDC skill definition into the agent configuration root.

Autonomous runtimes parse agent rules from designated local directories. For Claude Code, skills can be placed in ~/.claude/skills or .claude/skills; for Cursor and Windsurf, rules are placed in .cursor/rules or project roots. Creating this structured directory structure ensures seamless trigger detection across sessions.

Terminal Command

mkdir -p .agents/skills/debezium-cdc-change-data-capture && cat <<'EOF' > .agents/skills/debezium-cdc-change-data-capture/SKILL.md
---
name: "debezium-cdc-change-data-capture"
description: "Capture row-level database changes from PostgreSQL/MySQL into Kafka topics in real time."
metadata:
  author: "Debezium"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure debezium postgres connector"
  - "setup debezium cdc kafka"
  - "implement outbox pattern cdc"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Debezium CDC Database Replication to Kafka SOP
Follow strict postgresql.conf prerequisites, use pgoutput, validate via Connect REST API.
EOF

.agents/skills/debezium-cdc-change-data-capture/SKILL.md

---
name: "debezium-cdc-change-data-capture"
description: "Capture row-level database changes from PostgreSQL/MySQL into Kafka topics in real time."
metadata:
  author: "Debezium"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure debezium postgres connector"
  - "setup debezium cdc kafka"
  - "implement outbox pattern cdc"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

Production Pro-Tip

Keep the SKILL.md file lean by retaining execution rules and constraints while delegating large architecture diagrams to external documentation files to conserve LLM context tokens.

Sandbox Gotcha

Ensure the directory permissions allow the agent runtime read access. Avoid storing the skill outside the recognized agent search paths.

Database WAL Configuration & Connector JSON Formulation

Configure postgresql.conf for logical decoding and assemble the Kafka Connect JSON payload.

Debezium requires PostgreSQL to retain change records using write-ahead logging set to logical. The agent must construct a validated Kafka Connect configuration specifying io.debezium.connector.postgresql.PostgresConnector, setting plugin.name to pgoutput, and managing publication lifecycle.

Terminal Command

cat << 'EOF' > debezium-postgres-connector.json
{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "plugin.name": "pgoutput",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres_password",
    "database.dbname": "inventory_db",
    "database.server.name": "dbserver1",
    "table.include.list": "public.orders,public.outbox_events",
    "publication.autocreate.mode": "filtered",
    "tombstones.on.delete": "true",
    "decimal.handling.mode": "double"
  }
}
EOF

debezium-postgres-connector.json

{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "plugin.name": "pgoutput",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres_password",
    "database.dbname": "inventory_db",
    "database.server.name": "dbserver1",
    "table.include.list": "public.orders,public.outbox_events",
    "publication.autocreate.mode": "filtered",
    "tombstones.on.delete": "true"
  }
}

Production Pro-Tip

Set tombstones.on.delete to true if using Kafka log compaction so downstream consumer caches can safely evict deleted records.

Sandbox Gotcha

Never commit raw database passwords in source code. Use Kafka Connect config providers (e.g., FileConfigProvider) or workspace-scoped environment variables.

Intent Matching & Connector Registration

Trigger the agent with explicit natural language intents to deploy the connector via Kafka Connect REST API.

Provide the agent with explicit instructions matching the frontmatter triggers ('configure debezium postgres connector' or 'implement outbox pattern cdc'). The agent executes an idempotent curl request to create or update the connector config.

Terminal Command

curl -i -X PUT -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/inventory-connector/config -d @debezium-postgres-connector.json

Production Pro-Tip

Use PUT /connectors/{name}/config instead of POST /connectors to ensure deployment operations are idempotent and restart-safe.

Sandbox Gotcha

Ensure Kafka Connect has completed bootstrapping before triggering the registration command, or the HTTP request will fail with connection refused.

Verification, Replication Slot & Health Quality Gate

Inspect the connector status and verify database replication slot progression.

Agents must self-verify the connector state through automated status queries and inspect PostgreSQL to ensure replication slots are active and not accumulating unconsumed WAL logs.

Terminal Command

curl -s http://localhost:8083/connectors/inventory-connector/status | jq . && psql -h localhost -U postgres -d inventory_db -c "SELECT slot_name, plugin, active, wal_status FROM pg_replication_slots;"

Production Pro-Tip

Regularly monitor confirmed_flush_lsn and pg_current_wal_lsn() to preempt disk exhaustion caused by inactive consumers or paused connector tasks.

Sandbox Gotcha

If status shows FAILED, examine the trace property inside tasks[0] in the REST response rather than blindly restarting the container.

Real-World Scenario: Configure Reliable PostgreSQL CDC Outbox Pattern for Microservice Event Sourcing

Tested Trigger Prompt (Feed directly into Codex, Claude, or Cursor):

“Please configure Debezium Postgres connector for our e-commerce order service. We need to implement outbox pattern cdc from table public.outbox_events using pgoutput, route events to Kafka topic orders.events, and verify task status.”

Deterministic Agent Verification Pipeline:

Checks postgresql.conf settings (wal_level=logical, max_replication_slots >= 10, max_wal_senders >= 10) in PostgreSQL container.

Generates a valid connector configuration JSON specifying PostgresConnector, pgoutput plugin, and outbox event table filters.

Deploys the configuration via PUT /connectors/order-outbox-connector/config to Kafka Connect.

Queries GET /connectors/order-outbox-connector/status to ensure state is RUNNING and tests event emission by inserting a row into public.outbox_events.

Runtime Troubleshooting & Failure Modes

Error: org.postgresql.util.PSQLException: ERROR: logical decoding requires wal_level >= logical

Root Cause: The PostgreSQL instance was started with the default wal_level (replica or minimal), preventing Debezium from initializing the pgoutput logical replication slot.

Resolution Command / Fix:

ALTER SYSTEM SET wal_level = 'logical'; ALTER SYSTEM SET max_replication_slots = 10; ALTER SYSTEM SET max_wal_senders = 10; -- Restart PostgreSQL container/service afterwards: docker restart postgres

Error: io.debezium.DebeziumException: Creation of replication slot failed - slot "debezium" already exists

Root Cause: A previous connector crashed or was deleted without cleaning up its persistent replication slot, causing PostgreSQL to block creation of a slot with the identical name.

Resolution Command / Fix:

psql -U postgres -d inventory_db -c "SELECT pg_drop_replication_slot('debezium');"

Frequently Asked Questions (Debezium CDC Database Replication to Kafka SOP)

Q:Can I use Debezium CDC Database Replication to Kafka across Codex, Claude Code, and Cursor simultaneously?

Yes. The SKILL.md specification is platform-agnostic and fully functional across Codex, Claude Code, Cursor, and Windsurf by mounting the file into the respective agent skills or rules paths.

Q:How does Debezium CDC Database Replication to Kafka prevent context window overflow during long agent sessions?

The skill provides tight, deterministic SOP steps and explicit JSON templates rather than allowing the agent to repeatedly guess connection parameters and debug verbose error stacks across iterative failures.

Q:Does this skill require escalated root or container permissions?

No. The skill is audited with a security score of 98/100, requires no root host execution, and enforces safe, workspace-bounded file operations and standard HTTP API interactions with Kafka Connect.

Ready to equip your autonomous agent with Debezium CDC Database Replication to Kafka?

Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.

Get 1-Click Install Spec