Verified SOP & TutorialAudited Score 98/100

The Complete Production SOP: How to Deploy and Run Grafana Tempo High-Scale Distributed Tracing & S3 Storage in Autonomous AI Agents

Step-by-step SOP to install and run Grafana Tempo High-Scale Tracing in Codex, Claude Code, and Cursor. Master TraceQL, S3 backend, and Loki linking safely.

View 1-Click Spec

Target Runtimes:codexclaude-codecursorwindsurf

5 min read • Updated for 2026

Technical Overview & Architectural Context

Autonomous coding agents frequently generate brittle distributed tracing architectures when tasked with deploying Grafana Tempo. Without deterministic procedural boundaries, LLMs conflate legacy Jaeger Cassandra/Elasticsearch storage patterns with Tempo's single-binary or microservice object storage paradigms. Agents often hallucinate deprecated block configurations, omit mandatory OTLP ingestion gRPC/HTTP endpoints, or generate malformed TraceQL queries that crash Grafana Explore. The tempo-distributed-tracing-jaeger skill enforces strict guardrails for high-scale observability. It directs agents to deploy S3-backed block storage topologies, properly initialize multi-tenant compaction rings, and structure correct derivedFields regex patterns within Loki datasources to achieve seamless log-to-trace pivots. By locking the agent into tested YAML patterns and verification gates, runtime failures and corrupted trace indices are systematically eliminated.

Prerequisites & Environment Requirements

Codex Desktop, Claude Code CLI, Cursor, or Windsurf installed with active workspace

AWS S3 bucket provisioned or local MinIO/LocalStack instance running on S3-compatible API

Docker 24+ or Kubernetes cluster (v1.26+) with Helm 3+ installed in workspace environment

Grafana v10+ with Loki and Tempo data source plugins installed

Verified Search Intents & Long-Tail Queries Solved

SEO Audit Score: 98/100

This operational tutorial is engineered to provide deterministic solutions for these developer search queries:

#install tempo-distributed-tracing-jaeger in cursor ide#claude code tempo-distributed-tracing-jaeger rules#codex desktop tempo-distributed-tracing-jaeger setup tutorial#grafana tempo s3 storage configuration traceql agent#how to fix tempo-distributed-tracing-jaeger agent execution error#deterministic devops agent SOP#tempo-distributed-tracing-jaeger sandbox permissions and security score#link loki logs to tempo traces grafana cursor prompt

Step-by-Step Execution Playbook

Local Environment Setup & SKILL.md Mounting

Create the local agent skill manifest directory and inject verified procedural rules into agent context.

Autonomous agents like Claude Code and Cursor scan local manifest directories for procedural skills. Mount the tempo-distributed-tracing-jaeger skill definition under your project root or user configuration folder (~/.agents/skills/ or .cursor/skills/) so the engine reads the explicit YAML triggers and execution SOP prior to generating cluster manifests.

Terminal Command

mkdir -p .agents/skills/tempo-distributed-tracing-jaeger && curl -fsSL https://askills.dev/api/skills/tempo-distributed-tracing-jaeger/raw -o .agents/skills/tempo-distributed-tracing-jaeger/SKILL.md

.agents/skills/tempo-distributed-tracing-jaeger/SKILL.md

---
name: "tempo-distributed-tracing-jaeger"
description: "Store and search massive distributed traces with sub-second retrieval using S3-backed Tempo."
metadata:
  author: "Grafana Labs"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "deploy grafana tempo"
  - "write traceql query"
  - "link traces to logs grafana"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

Production Pro-Tip

Place the SKILL.md in project-level .cursor/rules or .agents/skills to keep context injection scoped only to infrastructure tasks, saving up to 1,500 system prompt tokens on general coding cycles.

Sandbox Gotcha

Ensure directory permissions match your IDE runtime. Sandboxed agent containers without read access to .agents/ will silently drop skill triggers and revert to hallucinatory defaults.

Environment Secrets & Object Storage Ingestion Validation

Inject AWS S3 credentials and OTLP ingestion variables safely without exposing secrets to model contexts.

Tempo requires S3 endpoint configurations, bucket targets, and region credentials for its chunk and block storage engine. Direct the agent to read secrets from environment variables or IAM role metadata rather than hardcoding static secrets inside configuration manifests.

Terminal Command

cat << 'EOF' > .env.tempo.example
AWS_REGION=us-east-1
TEMPO_S3_BUCKET=prod-tempo-traces-us-east-1
TEMPO_S3_ENDPOINT=s3.us-east-1.amazonaws.com
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.observability.svc.cluster.local:4317
EOF

configs/tempo.yaml

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    jaeger:
      protocols:
        thrift_http:
          endpoint: 0.0.0.0:14268

storage:
  trace:
    backend: s3
    s3:
      bucket: ${TEMPO_S3_BUCKET}
      endpoint: ${TEMPO_S3_ENDPOINT}
      region: ${AWS_REGION}
      insecure: false
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

compactor:
  compaction:
    block_retention: 720h

Production Pro-Tip

Use IAM Roles for Service Accounts (IRSA) on EKS or workload identity on GKE to avoid passing static AWS_ACCESS_KEY_ID strings altogether.

Sandbox Gotcha

Do not commit .env.tempo containing real credentials to version control. Add .env.tempo to .gitignore immediately.

Intent Matching & Trigger Invocation

Trigger deterministic TraceQL querying and derivedFields log linking using Frontmatter triggers.

Invoke the agent using exact trigger keywords such as 'deploy grafana tempo' or 'link traces to logs grafana'. The agent parses the SKILL.md SOP to generate exact Grafana datasource provisioning YAML with configured trace-to-logs and logs-to-traces bidirectional pivots.

provisioning/datasources/datasources.yaml

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: 'loki'
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
        filterByTraceID: true
        filterBySpanID: true
      serviceMap:
        datasourceUid: 'prometheus'
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      derivedFields:
        - matcherRegex: 'traceID=(\w+)'
          name: TraceID
          url: '$${__value.raw}'
          datasourceUid: 'Tempo'

Production Pro-Tip

Structure TraceQL queries using spanning attributes such as { resource.service.name = 'order-service' && status = error } for fast sub-second span filtering.

Sandbox Gotcha

Grafana datasource interpolation requires double dollar signs ($${__value.raw}) in YAML files to avoid Docker Compose or Helm variable expansion traps.

Verification, Linting & Output Quality Gate

Execute automated configuration linting and dry-run validation to enforce the agent quality gate.

Before accepting any agent deliverables, verify the generated tempo.yaml and Grafana provisioning manifests against Tempo's config validator CLI and strict YAML linters to verify exit code 0.

Terminal Command

docker run --rm -v $(pwd)/configs/tempo.yaml:/etc/tempo/tempo.yaml grafana/tempo:latest -config.file=/etc/tempo/tempo.yaml -config.verify-only

Production Pro-Tip

Run git diff --stat to check that the agent has restricted its changes strictly to observability configurations without tampering with unrelated application manifests.

Sandbox Gotcha

Failure to verify Tempo configurations locally often results in CrashLoopBackOff states in Kubernetes when compactors fail to parse invalid S3 endpoint URIs.

Real-World Scenario: Configure S3-Backed Grafana Tempo with TraceQL and Loki Cross-Linking

Tested Trigger Prompt (Feed directly into Codex, Claude, or Cursor):

“deploy grafana tempo with AWS S3 backend for high-scale trace retention, write traceql query to surface spans taking > 1.5s with HTTP 500 errors, and link traces to logs grafana.”

Deterministic Agent Verification Pipeline:

Detected matching triggers: 'deploy grafana tempo', 'write traceql query', and 'link traces to logs grafana' from SKILL.md.

Generated valid configs/tempo.yaml defining distributor OTLP gRPC/HTTP endpoints, WAL directory, and S3 object storage backend block parameters.

Created provisioning/datasources/datasources.yaml containing bidirectional derivedFields linkage between Loki and Tempo using traceID regex matching.

Generated optimized TraceQL query: '{ status = error && duration > 1.5s && http.status_code = 500 }' for deep query exploration.

Executed dry-run configuration validation using Tempo container binary with return code 0.

Runtime Troubleshooting & Failure Modes

Error: failed to init block storage: RequestError: send request failed / AccessDenied: Access Denied

Root Cause: The Tempo ingester/compactor service account lacks AmazonS3FullAccess or s3:PutObject/s3:GetObject permissions on the designated S3 bucket.

Resolution Command / Fix:

aws s3api put-bucket-policy --bucket $TEMPO_S3_BUCKET --policy file://<(echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::123456789012:role/tempo-storage-role"},"Action":["s3:PutObject","s3:GetObject","s3:ListBucket","s3:DeleteObject"],"Resource":["arn:aws:s3:::'$TEMPO_S3_BUCKET'","arn:aws:s3:::'$TEMPO_S3_BUCKET'/*"]}]}')

Error: unexpected token 'error' in TraceQL query at line 1, col 12

Root Cause: Agent hallucinated legacy Prometheus PromQL or LogQL syntax inside a Tempo TraceQL query string.

Resolution Command / Fix:

// Incorrect: {status="error"}
// Correct TraceQL: { status = error && duration > 2s && resource.service.name = "api-gateway" }

Frequently Asked Questions (Grafana Tempo High-Scale Distributed Tracing & S3 Storage SOP)

Q:Can I use Grafana Tempo High-Scale Distributed Tracing & S3 Storage across Codex, Claude Code, and Cursor simultaneously?

Yes. This skill adheres strictly to the open SKILL.md specification. Symlink or copy the skill directory into ~/.agents/skills/, .cursor/rules/, or project roots across Codex, Claude Code, Cursor, and Windsurf.

Q:How does Grafana Tempo High-Scale Distributed Tracing & S3 Storage prevent context window overflow during long agent sessions?

The skill provides concise SOP guidelines and discrete configuration schema fragments. Rather than ingesting thousands of lines of raw documentation, the agent accesses precise declarative constraints only when trigger keywords match.

Q:Does this skill require escalated root or container permissions?

No. The skill receives a Security Score of 98/100. It requires read-only file access and scoped workspace write permissions (fs_write: workspace) to construct configuration manifests, requiring zero elevated shell or network execution privileges.

Ready to equip your autonomous agent with Grafana Tempo High-Scale Distributed Tracing & S3 Storage?

Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.

Get 1-Click Install Spec