The Complete Production SOP: How to Deploy and Run OpenAI Realtime API WebRTC & Audio Sessions in Autonomous AI Agents

---
name: "openai-realtime-voice-webrtc"
description: "Build ultra-low-latency bidirectional voice agents with OpenAI Realtime API."
metadata:
  author: "OpenAI"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "connect to openai realtime api"
  - "stream audio to realtime api"
  - "handle realtime vad events"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# OpenAI Realtime API WebRTC & Audio Sessions

Implements OpenAI Realtime API (WebSocket / WebRTC): session.update, conversation.item.create, response.create, audio input/output streaming, and voice activity detection (VAD).

## When to Invoke
Invoke this skill whenever you need to:
- connect to openai realtime api
- stream audio to realtime api
- handle realtime vad events

## Execution SOP & Rules
1. Establish ephemeral session tokens via backend server to protect primary OpenAI API keys.
2. Configure audio format as PCM16 (24kHz) for optimal clarity and low transport latency.
3. Listen for input_audio_buffer.speech_started events to instantly cut off model audio playback.
4. Submit function call results promptly to avoid pausing the conversational audio flow.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.

export async function initRealtimeWebRTC(ephemeralKey: string) {
  const pc = new RTCPeerConnection();
  const audioEl = document.createElement('audio');
  audioEl.autoplay = true;

  pc.ontrack = (e) => {
    audioEl.srcObject = e.streams[0];
  };

  const ms = await navigator.mediaDevices.getUserMedia({ audio: { sampleRate: 24000, channelCount: 1 } });
  pc.addTrack(ms.getTracks()[0]);

  const dc = pc.createDataChannel('oai-events');
  dc.addEventListener('message', (event) => {
    const realtimeEvent = JSON.parse(event.data);
    if (realtimeEvent.type === 'input_audio_buffer.speech_started') {
      // Barge-in: Mute audio immediately while model cancels output
      audioEl.pause();
      audioEl.currentTime = 0;
    }
  });

  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);

  const baseUrl = 'https://api.openai.com/v1/realtime';
  const model = 'gpt-4o-realtime-preview-2024-12-17';
  const sdpResponse = await fetch(`${baseUrl}?model=${model}`, {
    method: 'POST',
    body: offer.sdp,
    headers: {
      Authorization: `Bearer ${ephemeralKey}`,
      'Content-Type': 'application/sdp'
    }
  });

  const answer = { type: 'answer' as RTCSdpType, sdp: await sdpResponse.text() };
  await pc.setRemoteDescription(answer);
  return { pc, dc };
}