The Complete Production SOP: How to Deploy and Run Portkey AI Gateway Load Balancing & Fallbacks in Autonomous AI Agents

---
name: "portkey-ai-gateway-proxy"
description: "Route LLM traffic through a fast, resilient AI Gateway with load balancing and retries."
metadata:
  author: "Portkey AI"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "setup portkey ai gateway"
  - "configure provider fallback portkey"
  - "load balance llm requests"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Portkey AI Gateway Load Balancing & Fallbacks

Configures Portkey AI Gateway: unified multi-provider routing (OpenAI, Anthropic, Bedrock), automated retries with exponential backoff, fallback provider failover, and request load balancing.

## When to Invoke
Invoke this skill whenever you need to:
- setup portkey ai gateway
- configure provider fallback portkey
- load balance llm requests

## Execution SOP & Rules
1. Deploy Portkey Gateway as a lightweight Docker container or serverless edge worker.
2. Configure routing configs declaring targets, retry attempts, and fallback provider sequences.
3. Route client SDK requests through the gateway by updating the baseURL parameter.
4. Monitor provider latency, error rates, and token throughput on centralized dashboards.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.

{
  "strategy": {
    "mode": "fallback"
  },
  "targets": [
    {
      "provider": "openai",
      "api_key": "${OPENAI_API_KEY}",
      "override_params": {
        "model": "gpt-4o"
      },
      "retry": {
        "attempts": 3,
        "on_status_codes": [429, 500, 503]
      }
    },
    {
      "provider": "anthropic",
      "api_key": "${ANTHROPIC_API_KEY}",
      "override_params": {
        "model": "claude-3-5-sonnet-20241022"
      }
    }
  ]
}

import OpenAI from 'openai';
import { createHeaders } from 'portkey-ai';

const portkeyConfig = require('../portkey-config.json');

export const aiClient = new OpenAI({
  apiKey: process.env.PORTKEY_API_KEY || 'dummy-key',
  baseURL: process.env.PORTKEY_GATEWAY_URL || 'http://localhost:8787/v1',
  defaultHeaders: createHeaders({
    config: portkeyConfig,
    virtualKey: 'openai-default-vk'
  })
});

import { aiClient } from '../src/llmClient';

describe('Portkey AI Gateway Routing', () => {
  it('successfully reaches gateway endpoint', async () => {
    expect(aiClient.baseURL).toContain(':8787/v1');
    expect(aiClient.defaultHeaders).toHaveProperty('x-portkey-config');
  });
});