The Complete Production SOP: How to Deploy and Run VictoriaMetrics High-Throughput Time Series Storage in Autonomous AI Agents
Production SOP to install and run VictoriaMetrics high-throughput time series storage in Codex, Claude Code, & Cursor. Includes remote_write configs & MetricsQL.
Technical Overview & Architectural Context
Autonomous coding agents frequently fail when provisioning telemetry pipelines due to outdated assumptions around Prometheus-native storage bottlenecks. When tasked with high-frequency telemetry architectures, agents commonly attempt unoptimized local TSDB deployments, configure inefficient WAL retention buffers, or produce invalid Prometheus remote_write blocks that trigger backpressure cascade failures. Additionally, agents hallucinate standard PromQL limitations when querying high-churn series, failing to exploit VictoriaMetrics-specific MetricsQL rollups and subqueries. The 'victoria-metrics-high-throughput' skill provides strict operational guardrails. It establishes deterministic component distribution across single-node binaries or distributed topologies (vmstorage, vminsert, and vmselect), calculates memory-to-cache operational ratios, generates compliant scrape and ingestion manifests, and translates naive PromQL queries into high-performance MetricsQL analytical functions. By scoping these constraints within the agent's procedural context, LLMs can autonomously generate production-grade telemetry architectures without crashing system memory or saturating IOPS.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local skill manifest directory and inject verified procedural rules.
Autonomous coding assistants require direct local access to the skill definition in their project or global workspace metadata directory. Claude Code, Cursor, and Codex automatically parse nested markdown skill manifests when initialized. Mount the skill inside the `.agents/skills` or `.claude/skills` directory so the runtime parser indexes the triggers and YAML boundary checks.
mkdir -p .agents/skills/victoria-metrics-high-throughput && curl -fsSL https://askills.dev/api/skills/victoria-metrics-high-throughput/raw -o .agents/skills/victoria-metrics-high-throughput/SKILL.md---
name: "victoria-metrics-high-throughput"
description: "Achieve 10x higher ingestion throughput and 7x better compression than standard Prometheus."
metadata:
author: "VictoriaMetrics"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "deploy victoriametrics cluster"
- "configure prometheus remote_write victoriametrics"
- "metricsql functions"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# VictoriaMetrics High-Throughput Time Series Storage SOP
- Prioritize single-node deployment (victoria-metrics-single) for <1M samples/sec.
- Split into vmstorage, vminsert, and vmselect when horizontal scale is required.
- Standard ingestion port: 8428 (single-node) or 8480 (vminsert).
- Always inject max_samples_per_send and send_exemplars false in remote_write unless explicitly required.Use `fs_write: "workspace"` to restrict agent modifications strictly to deployment manifests and Grafana dashboard JSONs without permitting global filesystem tampering.
Do not rename the folder arbitrarily; AI orchestrators correlate skill namespace triggers with folder paths in `.agents/skills/`.
Configuring Remote Ingestion & Cluster Manifests
Inject target endpoints and configure standard Prometheus remote_write schemas.
When prompted to wire Prometheus or vmagent into VictoriaMetrics, the agent must generate correct serialization and endpoint parameters. Single-node installations listen on `/api/v1/write`, while cluster installations direct writes through `vminsert` at `/insert/0/prometheus/api/v1/write`. The skill forces the agent to include queue configurations that guard against memory leaks during ingestion spikes.
cat << 'EOF' > prometheus-remote-write.yml
remote_write:
- url: "http://victoria-metrics:8428/api/v1/write"
queue_config:
max_samples_per_send: 10000
capacity: 50000
max_shards: 30
min_shards: 4
EOFremote_write:
- url: "http://victoria-metrics:8428/api/v1/write"
queue_config:
max_samples_per_send: 10000
capacity: 50000
max_shards: 30
min_shards: 4
write_relabel_configs:
- action: keep
source_labels: [__name__]
regex: "(http_requests_total|node_cpu_seconds_total|jvm_memory_used_bytes)"Enforce `write_relabel_configs` in the SOP to ensure the agent drops high-cardinality debugging metrics before they leave Prometheus.
Agents often append `/write` instead of `/api/v1/write` for single-node instances, which causes HTTP 404 remote-write failures.
Intent Matching & Trigger Invocation
Verify deterministic activation using tested natural language prompts.
Test that the target agent runtime (Claude Code, Cursor Composer, or Codex) indexes the skill and triggers the procedural SOP when provided domain-specific tasks. The agent will read frontmatter triggers like 'configure prometheus remote_write victoriametrics' and apply the embedded guidelines instead of generating non-deterministic code.
# Execute within Claude Code CLI or Cursor Composer
claude "configure prometheus remote_write victoriametrics for our k8s deployment targeting 500k samples per second"Use concise verbs matching the frontmatter triggers such as 'metricsql functions' to immediately force query synthesis into MetricsQL syntax.
If using Cursor or Windsurf, confirm that rules files (`.cursorrules` or `.windsurfrules`) point to or include `.agents/skills/victoria-metrics-high-throughput/SKILL.md`.
Verification, Linting & Output Quality Gate
Execute automated self-healing checks before accepting agent deliverables.
Instruct the agent to validate generated YAML configurations using `promtool` or VictoriaMetrics command-line utilities. If the agent generates MetricsQL queries, it must validate the syntax against the `/api/v1/query` endpoint or dry-run rules check.
promtool check config prometheus-remote-write.yml && helm lint ./helm/victoria-metrics-cluster#!/usr/bin/env bash
set -euo pipefail
if ! promtool check config prometheus-remote-write.yml; then
echo "Error: prometheus-remote-write.yml validation failed!"
exit 1
fi
echo "VictoriaMetrics ingestion configs passed validation."Review git diff with `git diff --staged` before accepting pull requests created by coding agents to ensure no unwanted scrape targets were stripped.
Running `promtool` against VictoriaMetrics-only MetricsQL functions like `rollup_changes()` will return syntax errors in standard promtool; make sure MetricsQL expressions are validated using the VictoriaMetrics binary directly.
Real-World Scenario: Migrating a Kubernetes Cluster Monitoring Stack to VictoriaMetrics High-Throughput
“We are experiencing out-of-memory crashes on our Prometheus instances due to 800,000 metrics/sec ingestion. Deploy a VictoriaMetrics cluster using Helm, set up remote_write forwarding from our Prometheus agents, and generate a MetricsQL query for the 95th percentile request duration grouped by service over a 15-minute rolling window.”
Runtime Troubleshooting & Failure Modes
Root Cause: The agent configured a cluster vminsert instance using the single-node path (`/api/v1/write`) instead of the multi-tenant cluster endpoint (`/insert/<accountID>/prometheus/api/v1/write`).
sed -i 's|:8480/api/v1/write|:8480/insert/0/prometheus/api/v1/write|g' prometheus-remote-write.ymlRoot Cause: Ingestion volume exceeds the default `max_shards` setting in Prometheus remote_write or `vmstorage` disk writes are IOPS-throttled.
yq eval '.remote_write[0].queue_config.max_shards = 64' -i prometheus-remote-write.yml && yq eval '.remote_write[0].queue_config.batch_send_deadline = "5s"' -i prometheus-remote-write.ymlFrequently Asked Questions (VictoriaMetrics High-Throughput Time Series Storage SOP)
Q:Can I use VictoriaMetrics High-Throughput Time Series Storage across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification supported across all modern agent runtimes. By placing the file in `.agents/skills/victoria-metrics-high-throughput/SKILL.md` or linking it in workspace configuration files, any compliant runtime will automatically read its triggers and boundary instructions.
Q:How does VictoriaMetrics High-Throughput Time Series Storage prevent context window overflow during long agent sessions?
The skill enforces structured declarative guidelines (such as Helm chart flags, remote_write queue configurations, and MetricsQL syntax tables) rather than flooding the LLM context with redundant source documentation. This minimizes token usage by up to 85% compared to pasting VictoriaMetrics docs into the prompt.
Q:Does this skill require escalated root or container permissions?
No. The skill is audited with a security score of 98/100, requiring only workspace file read/write permissions (`fs_read: true`, `fs_write: 'workspace'`). It requires no arbitrary shell execution (`shell_exec: false`) and does not initiate outbound network requests during agent evaluation.
Ready to equip your autonomous agent with VictoriaMetrics High-Throughput Time Series Storage?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.