The Complete Production SOP: How to Deploy and Run Thanos Highly Available Prometheus Long-Term Storage in Autonomous AI Agents
Deploy Thanos long-term S3 metrics storage using Codex, Claude Code, and Cursor. Configure sidecars, downsampling, compactor, and Querier deduplication.
Technical Overview & Architectural Context
Autonomous coding agents frequently fail when generating production Thanos architectures because they conflate Prometheus TSDB retention semantics with remote storage mechanisms. When asked to deploy Thanos, agents often mistakenly scale the Thanos Compactor to multiple replicas (which causes fatal TSDB block corruption and object storage races), omit critical Prometheus external labels required for query deduplication, or misconfigure S3 bucket storage credentials directly inside Pod environment variables rather than using secure secret mounts.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Mount the skill definition into your agent workspace to enforce strict architectural rules for Thanos components.
Agent runtimes such as Claude Code and Cursor scan local directories for `.claude/skills` or `.agents/skills`. By installing `thanos-long-term-metrics-storage`, the agent automatically receives deterministic instructions regarding single-replica Compactor constraints, Sidecar volume sharing, and Querier deduplication label requirements.
mkdir -p .agents/skills/thanos-long-term-metrics-storage && curl -fsSL https://askills.dev/api/skills/thanos-long-term-metrics-storage/raw -o .agents/skills/thanos-long-term-metrics-storage/SKILL.mdtype: S3
config:
bucket: "prod-prometheus-thanos-metrics"
endpoint: "s3.us-west-2.amazonaws.com"
insecure: false
signature_version2: false
put_user_metadata:
owner: "monitoring"
http_config:
idle_conn_timeout: 90s
response_header_timeout: 2m
insecure_skip_verify: falseInstruct the agent to reference `thanos-storage-config.yaml` as an external Secret volume mount to avoid exposing raw credentials in manifests.
Do not grant shell execution permissions if your agent environment only requires manifest generation; keep `fs_write: workspace` strictly bounded.
Environment Secrets & Object Storage Configuration
Construct the Kubernetes Secret for Thanos S3 access and configure Prometheus external labels.
Thanos Sidecar and Store Gateway require a unified `thanos.yaml` object storage configuration file. The agent must inject this as a Kubernetes Secret and ensure Prometheus has unique `replica` and `cluster` external labels so Thanos Query can perform deduplication.
kubectl create secret generic thanos-objstore-secret --from-file=thanos.yaml=./thanos-storage-config.yaml -n monitoring --dry-run=client -o yaml > manifests/thanos-secret.yamlspec:
replicas: 2
thanos:
version: v0.34.1
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-secret
externalLabels:
cluster: "production-us-west-2"
replica: "$(POD_NAME)"Using dynamic pod names for the replica label ensures Thanos Query can identify and deduplicate queries across high-availability pairs.
Failing to set distinct externalLabels will cause Thanos Query to return duplicate metric samples across HA Prometheus pods.
Intent Matching & Component Deployment
Prompt the agent with specific triggers to generate Thanos Store Gateway, Compactor, and Query manifests.
Invoke the skill using explicit trigger terms. The agent will execute step-by-step manifest creation, applying downsampling retention rules (5m and 1h resolutions) on the Compactor while enforcing a replica count of exactly 1.
claude 'deploy thanos sidecar and thanos compactor downsampling for our monitoring namespace'apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-compactor
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: thanos-compactor
template:
metadata:
labels:
app.kubernetes.io/name: thanos-compactor
spec:
containers:
- name: compactor
image: quay.io/thanos/thanos:v0.34.1
args:
- "compact"
- "--data-dir=/var/thanos/compact"
- "--objstore.config-file=/etc/thanos/thanos.yaml"
- "--retention.resolution-raw=30d"
- "--retention.resolution-5m=180d"
- "--retention.resolution-1h=3y"
- "--wait"
volumeMounts:
- name: config
mountPath: /etc/thanos
- name: data
mountPath: /var/thanos/compact
volumes:
- name: config
secret:
secretName: thanos-objstore-secret
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50GiAlways set `--wait` on the Thanos Compactor to keep the process continuously looping rather than terminating as an ephemeral Job.
Never increase compactor replicas above 1; Thanos Compactor does not support distributed locking on S3 buckets and causes data corruption if run concurrently.
Verification, Linting & Output Quality Gate
Validate generated Kubernetes manifests using kubeval, kubeconform, or kubectl dry-run.
The skill enforces automated self-correction. The agent must dry-run all generated manifests against the local Kubernetes API or schema validator before reporting completion.
kubectl apply --dry-run=client -k manifests/ && kubeconform -strict manifests/#!/usr/bin/env bash
set -euo pipefail
echo "Checking Prometheus external labels..."
kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.externalLabels}'
echo "Validating Thanos Store endpoints..."
kubectl logs -n monitoring -l app.kubernetes.io/name=thanos-query -c query --tail=50 | grep -i "registered new store"Include the verification shell script in the agent pull request so CI/CD pipelines can replay the deterministic validation check.
Make sure the Thanos Query `--query.replica-label` flag matches the Prometheus `replica` external label exactly.
Real-World Scenario: Configure Long-Term S3 Retention with Thanos Query and Compactor Downsampling
“Configure Thanos store S3 and deploy Thanos compactor downsampling for our multi-cluster Prometheus setup. Ensure Thanos Querier deduplicates metrics across our two Prometheus replicas.”
Runtime Troubleshooting & Failure Modes
Root Cause: Multiple Thanos Compactor instances ran concurrently against the same S3 prefix, or Prometheus instances uploaded TSDB blocks with identical external labels without replica deduplication flags.
kubectl scale statefulset thanos-compactor -n monitoring --replicas=0 && kubectl delete pod thanos-compactor-0 -n monitoring && thanos tools bucket verify --objstore.config-file=/etc/thanos/thanos.yaml --repair && kubectl scale statefulset thanos-compactor -n monitoring --replicas=1Root Cause: Thanos Query or Store Gateway hit default gRPC max message size limits (4MB) during large multi-month historical range scans.
Add flag `--grpc-client-max-receive-message-size=104857600` (100MB) to Thanos Query container args and `--grpc-server-max-send-message-size=104857600` to Thanos Store Gateway args.Frequently Asked Questions (Thanos Highly Available Prometheus Long-Term Storage SOP)
Q:Can I use Thanos Highly Available Prometheus Long-Term Storage across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification. It can be installed into `.agents/skills` or `.claude/skills` and read seamlessly by Codex CLI, Claude Code, Cursor, and Windsurf.
Q:How does Thanos Highly Available Prometheus Long-Term Storage prevent context window overflow during long agent sessions?
By providing concise, modular manifests and targeted verification steps rather than dumping thousands of lines of unneeded Helm values into the context window, preserving token limits for actionable infrastructure code.
Q:Does this skill require escalated root or container permissions?
No. The skill is audited with a security score of 98/100. It requires only standard workspace filesystem write permissions (`fs_write: workspace`) to generate Kubernetes manifests and execute non-root client validation checks.
Ready to equip your autonomous agent with Thanos Highly Available Prometheus Long-Term Storage?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.