Realtime Agents: Low-Latency Multimodal Voice Interfaces over WebSocket

Voice is the most natural interface for humans. But building a voice agent that feels responsive—where the user can interrupt, where the model responds in real time, and where tools can be invoked mid-conversation—requires more than chaining STT to LLM to TTS.

The OpenAI Agents SDK introduces Realtime Agents: server-side, low-latency agents built directly on the OpenAI Realtime API over WebSocket transport. These agents handle live audio, semantic turn detection, and tool execution in a single session, all managed by the familiar Agent and Runner abstractions.

In this tutorial, you will learn how to set up a server-side realtime session, configure audio input/output, and handle events from a live voice agent.

Prerequisites

• Python 3.10 or higher
• OpenAI API key
• Basic familiarity with the OpenAI Agents SDK

pip install openai-agents

The Realtime Architecture

Unlike the chained voice pipeline approach, a realtime agent processes audio as a continuous stream. The model receives raw PCM audio and produces audio output directly, with built-in voice activity detection and interruption handling.

Architecture	Latency	Control	Best for
Chained pipeline	~1-2s	Full (explicit STT → LLM → TTS)	Predictable workflows, extending text agents
Realtime agent	<500ms	Model-managed audio stream	Natural conversation, barge-in support

The key difference: the model itself handles the audio stream, including when to listen, when to respond, and how to handle interruptions.

Setting Up a Realtime Session

Step 1: Define the Agent

Start with a RealtimeAgent instead of a regular Agent. The instructions should be concise and conversational since the output is spoken:

from agents.realtime import RealtimeAgent, RealtimeRunner

agent = RealtimeAgent(
    name="Assistant",
    instructions="You are a helpful voice assistant. Keep responses short and conversational.",
)

Step 2: Configure the Runner

The runner configuration specifies the model and audio settings. Use the nested audio.input / audio.output structure for new code:

runner = RealtimeRunner(
    starting_agent=agent,
    config={
        "model_settings": {
            "model_name": "gpt-realtime-2.1",
            "audio": {
                "input": {
                    "format": "pcm16",
                    "transcription": {"model": "gpt-4o-mini-transcribe"},
                    "turn_detection": {
                        "type": "semantic_vad",
                        "interrupt_response": True,
                    },
                },
                "output": {
                    "format": "pcm16",
                    "voice": "ash",
                },
            },
        }
    },
)

Key settings to configure:

• model_name: Use gpt-realtime-2.1 for new agents
• audio.input.format: pcm16 for raw audio
• turn_detection.type: semantic_vad for intelligent turn detection
• interrupt_response: Enable barge-in support
• audio.output.voice: Choose from available voices (e.g., ash, alloy, echo)

Step 3: Run the Session

The RealtimeSession manages the WebSocket connection. Send messages and stream events:

import asyncio
from agents.realtime import RealtimeAgent, RealtimeRunner

async def main() -> None:
    agent = RealtimeAgent(
        name="Assistant",
        instructions="You are a helpful voice assistant. Keep responses short and conversational.",
    )

    runner = RealtimeRunner(
        starting_agent=agent,
        config={
            "model_settings": {
                "model_name": "gpt-realtime-2.1",
                "audio": {
                    "input": {
                        "format": "pcm16",
                        "transcription": {"model": "gpt-4o-mini-transcribe"},
                        "turn_detection": {
                            "type": "semantic_vad",
                            "interrupt_response": True,
                        },
                    },
                    "output": {
                        "format": "pcm16",
                        "voice": "ash",
                    },
                },
            }
        },
    )

    session = await runner.run()

    async with session:
        await session.send_message("Say hello in one short sentence.")

        async for event in session:
            if event.type == "audio":
                # Forward or play event.audio.data
                pass
            elif event.type == "history_added":
                print(event.item)
            elif event.type == "agent_end":
                # One assistant turn finished
                break
            elif event.type == "error":
                print(f"Error: {event.error}")

if __name__ == "__main__":
    asyncio.run(main())

Step 4: Sending Audio

For raw audio input instead of text messages, use send_audio():

# Send raw PCM audio chunks
await session.send_audio(audio_bytes)

# Or send a text message
await session.send_message("What's the weather like?")

Semantic Voice Activity Detection

The semantic_vad turn detection uses the model to determine when the user has finished speaking, rather than relying on silence thresholds. This enables natural conversation flow with:

• Automatic turn-taking: The model detects when the user stops speaking
• Barge-in support: The user can interrupt mid-response
• Contextual awareness: The model understands speech patterns, not just audio levels

"turn_detection": {
    "type": "semantic_vad",
    "interrupt_response": True,  # Allow user to interrupt
}

Tool Execution in Realtime

Realtime agents can call tools during conversation. The model decides when to invoke a tool based on the user’s request, executes it, and continues the voice interaction:

from agents import Agent, function_tool

@function_tool
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"The weather in {city} is 72°F and sunny."

agent = RealtimeAgent(
    name="Weather Assistant",
    instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
    tools=[get_weather],
)

The model will invoke the tool when appropriate and speak the result back to the user.

Connection Options

Environment Variables

Set your API key in the environment:

export OPENAI_API_KEY="your-api-key-here"

Custom Endpoints

For Azure OpenAI or custom deployments:

session = await runner.run(model_config={
    "url": "wss://your-endpoint.openai.azure.com/realtime",
    "headers": {"Authorization": "Bearer your-key"},
})

Existing Calls

Attach to an existing realtime call:

session = await runner.run(model_config={
    "call_id": "existing-call-id",
})

Key Settings Reference

Setting	Description	Default
model_name	Realtime model to use	gpt-realtime-2.1
audio.input.format	Input audio format	pcm16
audio.output.format	Output audio format	pcm16
audio.input.transcription	STT model configuration	gpt-4o-mini-transcribe
audio.input.turn_detection	VAD configuration	semantic_vad
audio.output.voice	Output voice	ash
tool_choice	Tool selection strategy	auto

Production Considerations

• Audio buffering: Manage PCM chunk sizes for your network conditions
• Error handling: Implement retry logic for WebSocket disconnections
• Session state: Use the Sessions API for persistent conversations
• Monitoring: Integrate tracing for debugging production voice agents

Next Steps

• Explore Realtime Transport to choose between server-side WebSocket and SIP
• Read the Realtime Agents Guide for lifecycle management, structured input, and guardrails
• Browse the examples for complete working implementations

Building realtime voice agents is a fundamental shift from text-based interactions. The OpenAI Agents SDK makes this accessible by extending the same Agent and Runner patterns you already know into the voice domain, giving you low-latency, multimodal conversation with full tool execution capabilities.