The Complete Production SOP: How to Deploy and Run Apache Pulsar Multi-Tenant Streaming & Tiered Storage in Autonomous AI Agents
Step-by-step SOP to install and run Apache Pulsar S3 tiered storage skill in Codex, Claude Code, and Cursor. Includes verified prompts, configs, and fixes.
Technical Overview & Architectural Context
Apache Pulsar separates compute (stateless broker layer) from storage (stateful Apache BookKeeper bookies), enabling granular multi-tenancy and transparent historical data offload to cloud storage. When autonomous AI agents are tasked with configuring Pulsar environments, they routinely hallucinate monolithic Kafka-style properties, misconfigure AWS S3 offloader drivers, fail to map hierarchical tenant/namespace/topic structures, or select inappropriate subscription models (Exclusive vs Failover vs Key_Shared). These mistakes result in unevicted BookKeeper ledger disk exhaustion, broken ledger segment offloading, and silent consumer bottlenecks. Without a deterministic procedural boundary, agents often attempt direct write modifications to deployed Bookie ledgers or fail to set namespace-level offload thresholds properly. The apache-pulsar-tiered-storage-s3 skill constrains the agent to authoritative operational protocols, ensuring deterministic pulsar-admin CLI calls, correct conf/tiered-storage-s3.conf parameter definitions, and explicit verification loops. By enforcing strict parameter ranges for ledger segment roll thresholds and S3 IAM offloader authentication, the skill prevents infrastructure corruption and ledger desynchronization across multi-cloud streaming deployments.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local skill manifest directory and inject verified procedural rules.
Autonomous coding agents discover capability boundaries by parsing the SKILL.md specification located in project-level or global agent directories. For Claude Code, Codex, and Cursor, placing the specification within .agents/skills/apache-pulsar-tiered-storage-s3/ or .cursor/skills/ exposes the triggers and operational guardrails to the planner before tool execution. The runtime reads the YAML frontmatter to index triggers such as 'configure pulsar s3 offloader' and 'setup pulsar subscription modes'.
mkdir -p .agents/skills/apache-pulsar-tiered-storage-s3 && cat <<'EOF' > .agents/skills/apache-pulsar-tiered-storage-s3/SKILL.md
---
name: "apache-pulsar-tiered-storage-s3"
description: "Decouple stream serving from storage and tier historical topic logs to cloud object storage."
metadata:
author: "Apache Software Foundation"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "configure pulsar s3 offloader"
- "setup pulsar subscription modes"
- "deploy pulsar functions"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# Apache Pulsar Multi-Tenant Streaming & Tiered Storage SOP
- Configure conf/tiered-storage-s3.conf with AWS credentials and target S3 bucket.
- Set namespace offload policy: bin/pulsar-admin namespaces set-offload-threshold --size 1G my-tenant/my-ns.
- Subscribe using ConsumerType: Exclusive, Shared, Failover, or Key_Shared.
- Trigger manual offload: bin/pulsar-admin topics offload --size-threshold 10M persistent://tenant/ns/topic.
EOF---
name: "apache-pulsar-tiered-storage-s3"
description: "Decouple stream serving from storage and tier historical topic logs to cloud object storage."
metadata:
author: "Apache Software Foundation"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "configure pulsar s3 offloader"
- "setup pulsar subscription modes"
- "deploy pulsar functions"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---Place the skill at .agents/skills/ rather than global root to ensure project-specific repository check-in and prevent cross-project context leakage.
Ensure fs_write is strictly constrained to workspace so the agent does not attempt to overwrite broker configuration files outside of the target repository.
Tiered Storage Configuration & IAM Validation
Inject required environment variables safely without leaking secrets into conversation history.
Pulsar uses the JCloud S3 offloader driver package to stream segment ledgers from BookKeeper out to S3. To prevent agent token exposure, credentials and bucket configurations should be managed via environment variables and loaded into broker offloader properties. The agent must create conf/tiered-storage-s3.conf referencing standard environment variables or instance metadata credentials rather than hardcoding static access tokens.
cat <<'EOF' > conf/tiered-storage-s3.conf
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadBucket=company-pulsar-tiered-storage-prod
s3ManagedLedgerOffloadServiceEndpoint=
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
EOFLeverage AWS DefaultAWSCredentialsProviderChain in production rather than explicit keys to allow automatic token rotation via IAM roles.
Ensure conf/tiered-storage-s3.conf or any local env files containing staging test keys are listed in .gitignore before prompt invocation.
Intent Matching & Trigger Invocation
Verify deterministic activation using tested natural language prompts.
Trigger the agent using the exact natural language anchors mapped in the SKILL.md definition. When issuing requests such as 'configure pulsar s3 offloader' or 'setup pulsar subscription modes', the agent runtime matches the frontmatter triggers and adopts the structured SOP, executing the namespace configuration and subscription validation steps deterministically.
pulsar-admin namespaces set-offload-threshold --size 1G finance-tenant/transactionsUse concise verbs matching the frontmatter triggers ('configure pulsar s3 offloader') to guarantee 100% routing fidelity into this skill pipeline.
Do not mix Kafka command syntax (e.g., kafka-topics.sh) in prompts, as this might confuse the agent's intent-matching heuristic.
Verification, Linting & Output Quality Gate
Execute automated self-healing checks before accepting agent deliverables.
The agent must self-verify its actions by executing dry-run tests or query checks against the Pulsar administrative REST API or CLI. The quality gate verifies that the offload threshold is set, that the driver is acknowledged by the broker, and that topic subscription statuses reflect the target delivery semantics.
pulsar-admin namespaces get-offload-threshold finance-tenant/transactions && pulsar-admin topics offload-status persistent://finance-tenant/transactions/settlement-eventsRun 'git diff' to review configuration property modifications before allowing the agent to commit broker changes.
Manual offloading will fail with an error if the topic's active ledger has not closed yet; ensure size thresholds exceed current ledger roll sizes.
Real-World Scenario: Configure S3 Tiered Storage and Key_Shared Consumer Subscription for Financial Event Topic
“configure pulsar s3 offloader for namespace finance-tenant/transactions with a 1GB threshold using bucket company-pulsar-tiered-storage-prod, and generate a Python consumer client script using Key_Shared subscription mode to process persistent://finance-tenant/transactions/settlement-events without out-of-order deliveries per key.”
Runtime Troubleshooting & Failure Modes
Root Cause: The IAM role or credentials mounted to the broker bookie lack s3:PutObject, s3:AbortMultipartUpload, or s3:ListBucket permissions on the target tiered storage S3 bucket.
aws iam put-role-policy --role-name PulsarBrokerRole --policy-name S3TieredStoragePolicy --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:AbortMultipartUpload","s3:ListBucket"],"Resource":["arn:aws:s3:::company-pulsar-tiered-storage-prod","arn:aws:s3:::company-pulsar-tiered-storage-prod/*"]}]}'Root Cause: Producer messages were published without a message routing key (MessageRoutingKey/OrderingKey), preventing Key_Shared deterministic hashing.
# Update producer code to enforce ordering keys on all emitted messages:
producer.send('transaction-payload'.encode('utf-8'), partition_key='account_id_49204')Frequently Asked Questions (Apache Pulsar Multi-Tenant Streaming & Tiered Storage SOP)
Q:Can I use Apache Pulsar Multi-Tenant Streaming & Tiered Storage across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification supported across all modern agent runtimes. By placing the file in .agents/skills/apache-pulsar-tiered-storage-s3/SKILL.md, Codex Desktop, Claude Code CLI, Cursor, and Windsurf can simultaneously load procedural boundaries and run verified Pulsar admin commands.
Q:How does Apache Pulsar Multi-Tenant Streaming & Tiered Storage prevent context window overflow during long agent sessions?
By establishing concise procedural boundaries and deterministic validation gates rather than polluting system prompts with redundant rules. The SKILL.md file abstracts complex BookKeeper and JCloud driver internals into precise triggers and step-by-step administrative actions, keeping context usage under 350 tokens.
Q:Does this skill require escalated root or container permissions?
No. It is audited with a security score of 98/100 and runs safely within workspace-scoped sandboxes. The skill operates exclusively via standard pulsar-admin CLI calls, declarative property configuration files, and standard Pulsar language client libraries without needing root access.
Ready to equip your autonomous agent with Apache Pulsar Multi-Tenant Streaming & Tiered Storage?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.