The Complete Production SOP: How to Deploy and Run Kubernetes HPA Custom Metrics & Prometheus Adapter in Autonomous AI Agents
Production SOP to install and run Kubernetes HPA Custom Metrics & Prometheus Adapter in Codex, Claude Code, and Cursor with verified prompts and fixes.
Technical Overview & Architectural Context
Autonomous coding agents (Codex, Claude Code, Cursor, Windsurf) frequently fail when configuring Kubernetes autoscaling with Prometheus Adapter. LLMs routinely hallucinate deprecated HorizontalPodAutoscaler schemas (such as legacy `autoscaling/v1` or early `v2beta1` definitions) that lack structural support for modern `behavior` controls, custom pod metrics, or external object references. Furthermore, agents often misconfigure Prometheus Adapter discovery rules by failing to properly define series queries, Prometheus namespace-pod metric associations, or resource label relabeling, leading to the infamous `unable to fetch pod metrics for ...: no metrics returned from custom metrics API` failure. Without explicit deterministic procedures, agents attempt naive trials that cycle through cluster-level RBAC adjustments without addressing underlying metric discovery mismatches. The `kubernetes-hpa-custom-metrics` skill enforces strict schema boundaries: it restricts outputs to `autoscaling/v2`, mandates Prometheus Adapter configuration via tested discovery rules, verifies metric routing through `custom.metrics.k8s.io/v1beta1`, and configures anti-flapping stabilization windows. This eliminates agent trial-and-error, saves context tokens, and protects cluster stability.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local skill manifest directory and inject verified procedural rules into the agent runtime.
Autonomous coding agents require local filesystem-level declarative manifests to activate runtime skills. By mounting the skill to `~/.agents/skills/kubernetes-hpa-custom-metrics/SKILL.md` or the workspace-level `.cursor/rules` / `.claude/skills` directory, agents automatically parse frontmatter triggers such as 'configure k8s hpa v2' or 'autoscale on http requests prometheus' and load deterministic SOP steps directly into execution context.
mkdir -p ~/.agents/skills/kubernetes-hpa-custom-metrics && curl -fsSL https://askills.dev/api/skills/kubernetes-hpa-custom-metrics/raw -o ~/.agents/skills/kubernetes-hpa-custom-metrics/SKILL.md---
name: "kubernetes-hpa-custom-metrics"
description: "Autoscale Kubernetes pods horizontally based on custom Prometheus business metrics."
metadata:
author: "Kubernetes"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "configure k8s hpa v2"
- "autoscale on http requests prometheus"
- "setup prometheus adapter hpa"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---Instruct Claude Code or Cursor to read SKILL.md on-demand during trigger invocation rather than pasting the raw manifest into the global system prompt to save over 1,500 context tokens per prompt roundtrip.
Ensure that the agent sandbox has workspace filesystem write permissions (`fs_write: workspace`); otherwise, attempts to output Helm values files or YAML manifests will fail silently or abort the run.
Environment Secrets & Dependency Validation
Validate cluster connectivity, Prometheus URL endpoint, and Helm repositories safely without exposing cluster secrets.
Before the agent generates Prometheus Adapter configurations, it must identify the target Prometheus server address and ensure Helm repositories are reachable. Environment parameters like Prometheus endpoints and target namespace must be staged via `.env.local` or standard shell exports so agents don't fabricate arbitrary cluster service endpoints.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update && kubectl get svc -A | grep prometheusUse internal Kubernetes cluster DNS (such as `http://prometheus-operated.monitoring.svc.cluster.local:9090`) to prevent unnecessary external network calls and egress firewall issues.
Never feed full kubeconfig certificates or authentication bearer tokens into LLM prompts; reference standard KUBECONFIG file paths instead.
Intent Matching & Trigger Invocation
Trigger the agent to generate Prometheus Adapter discovery rules and HPA v2 definitions.
Provide a clean, declarative trigger prompt matching the SKILL.md definition. The agent will read the procedural boundaries, structure the custom metric transformation query inside Prometheus Adapter's `rules.custom` section, and construct an `autoscaling/v2` HorizontalPodAutoscaler spec.
prometheus:
url: http://prometheus-k8s.monitoring.svc
port: 9090
rules:
custom:
- seriesQuery: '{__name__=~"^http_requests_total$",namespace!="",pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: "^http_requests_total$"
as: "http_requests_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'Always require the agent to specify the `as` alias in the Prometheus Adapter rule to prevent invalid characters in Kubernetes custom metric API endpoints.
Failing to set `by (<<.GroupBy>>)` in `metricsQuery` prevents the adapter from associating individual pod metrics back to the HorizontalPodAutoscaler target.
Verification, Linting & Output Quality Gate
Execute automated validation checks against custom metric endpoints and manifest schemas before completing the task.
The skill mandates running local schema validation and raw API querying. The agent must verify that the generated HPA manifest validates under Kubernetes OpenAPI schemas and query the custom metrics aggregation API endpoint to confirm metric discovery.
kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1' | jq '.resources[] | select(.name=="pods/http_requests_per_second")' && kubectl apply --dry-run=client -f hpa.yamlCheck git diff with `git diff --stat` before confirming agent completion to verify that unwanted namespace-level cluster role changes were not generated.
The custom metrics API returns an HTTP 404 until the Prometheus Adapter has completed its first discovery loop (typically 30–60 seconds after deployment).
Real-World Scenario: Autoscaling an API Gateway Deployment based on HTTP Requests per Second
“configure k8s hpa v2 to autoscale on http requests prometheus for payment-service in payments namespace. Scale up when requests exceed 150 rps per pod, with a 5-minute scale-down stabilization window.”
Runtime Troubleshooting & Failure Modes
Root Cause: The Prometheus Adapter rule discovery query failed to map the Prometheus 'pod' label to the Kubernetes Pod resource, or Prometheus scrape interval has not completed.
kubectl describe custommetric pods/http_requests_per_second -n payments && kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-adapter --tail=50Root Cause: The HPA spec is targeting a metric source `type: Pods` but the metric is exposed at deployment or ingress object level, or metric name does not match the adapter alias.
yq eval '.spec.metrics[0].pods.metric.name' hpa.yaml && kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/payments/pods/*/http_requests_per_second'Frequently Asked Questions (Kubernetes HPA Custom Metrics & Prometheus Adapter SOP)
Q:Can I use Kubernetes HPA Custom Metrics & Prometheus Adapter across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification. You can symlink or copy the SKILL.md manifest into `~/.agents/skills/kubernetes-hpa-custom-metrics/` or point your workspace rules directly to it in Codex, Claude Code CLI, Cursor, and Windsurf.
Q:How does Kubernetes HPA Custom Metrics & Prometheus Adapter prevent context window overflow during long agent sessions?
By establishing concise procedural boundaries and deterministic validation gates, the agent does not guess API versions or waste LLM context with multi-turn debugging of invalid `autoscaling/v2` schemas and Prometheus Adapter discovery syntax.
Q:Does this skill require escalated root or container permissions?
No. The skill is audited with a security score of 98/100. It requires only standard workspace filesystem write permissions (`fs_write: workspace`) to create Helm configuration files and Kubernetes manifests; it enforces `shell_exec: false` and `net_access: false` within the agent frontmatter.
Ready to equip your autonomous agent with Kubernetes HPA Custom Metrics & Prometheus Adapter?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.