Voice Agents: Building Speech-to-Text to Agent to TTS Pipelines

Voice is the interface people actually have with machines. And here’s the good news: you don’t need to redesign your agent to make it speak. The same Agent that answers text questions can power a full voice assistant, with speech-to-text on the front and text-to-speech on the back.

The OpenAI Agents SDK treats voice as a wrapper around the agent loop. A VoicePipeline does three steps: transcribe audio to text, run your (existing) agent workflow, and turn the result back into audio. Text agent in, voice agent out—in a few lines.

In this tutorial, you will learn the two voice architectures, how to build a chained voice pipeline, and how to tune the personality of the spoken output.

Prerequisites

• Python 3.9+ with pip install 'openai-agents[voice]'
• An OpenAI API key
• sounddevice and numpy for microphone/speaker I/O

Two Voice Architectures

The first design decision is whether the model works with live audio or whether your app chains explicit stages:

Architecture	Best for	Why
Speech-to-speech (realtime)	Natural, low-latency conversations	The model handles live audio directly: barge-in, fast first audio, natural turn-taking
Chained voice pipeline	Predictable workflows, extending existing text agents	Your app keeps explicit control over transcription, reasoning, and speech

The chained path is the natural extension of a text agent you already have, and it’s the simplest to start with.

The VoicePipeline

VoicePipeline is a three-step process:

1. Run a speech-to-text model to turn audio into text.
2. Run your code—usually an agentic workflow—to produce a result.
3. Run a text-to-speech model to turn the result back into audio.

from agents import Agent
from agents.voice import (
    AudioInput,
    SingleAgentVoiceWorkflow,
    VoicePipeline,
)

agent = Agent(
    name="Assistant",
    instructions="You are a helpful voice assistant.",
    tools=[get_weather],
)

pipeline = VoicePipeline(workflow=SingleAgentVoiceWorkflow(agent))

SingleAgentVoiceWorkflow wraps your existing agent. The voice surface changes the transport and audio loop; the core workflow decisions stay the same.

Running the Pipeline

Feed it an audio buffer and stream the spoken response:

import numpy as np
import sounddevice as sd
from agents.voice import AudioInput

# For simplicity, 3 seconds of silence. In reality, microphone data.
buffer = np.zeros(24000 * 3, dtype=np.int16)
audio_input = AudioInput(buffer=buffer)

result = await pipeline.run(audio_input)

player = sd.OutputStream(samplerate=24000, channels=1, dtype=np.int16)
player.start()

async for event in result.stream():
    if event.type == "voice_stream_event_audio":
        player.write(event.data)
    elif event.type == "voice_stream_event_lifecycle":
        # turn_started / turn_ended — use these to mute/unmute the mic
        print("lifecycle:", event.lifecycle_event)
    elif event.type == "voice_stream_event_error":
        print("error:", event.error)

Two input modes exist:

• AudioInput — a complete audio buffer. Use it for pre-recorded audio or push-to-talk, where it’s clear when the user is done.
• StreamedAudioInput — push chunks as they’re detected; the pipeline uses activity detection to run the workflow automatically when the user stops speaking.

Streamed Microphone Input

For hands-free conversations, stream from the microphone and let the pipeline detect turn boundaries:

from agents.voice import StreamedAudioInput

stream = StreamedAudioInput()

async def capture_mic():
    with sd.InputStream(samplerate=24000, channels=1, dtype="int16") as mic:
        while True:
            chunk, _ = mic.read(480)   # ~20ms
            await stream.add_audio(chunk)

result = await pipeline.run(stream)

Note: the SDK doesn’t provide built-in interruption handling for StreamedAudioInput—every detected turn triggers a separate workflow run. Listen to turn_started/turn_ended lifecycle events to mute the speaker’s microphone while the model talks.

Tuning the Spoken Personality

The default TTS model (gpt-4o-mini-tts) gives you huge control over the voice via TTSModelSettings:

from agents.voice import (
    TTSModelSettings,
    VoicePipelineConfig,
)

custom_tts_settings = TTSModelSettings(
    instructions=(
        "Personality: upbeat, friendly, persuasive guide. "
        "Tone: Friendly, clear, and reassuring, creating a calm atmosphere "
        "and making the listener feel confident and comfortable. "
        "Pronunciation: Clear, articulate, and steady, ensuring each "
        "instruction is easily understood while maintaining a natural, "
        "conversational flow. "
        "Tempo: Speak relatively fast, include brief pauses after questions. "
        "Emotion: Warm and supportive, conveying empathy and care."
    )
)

pipeline = VoicePipeline(
    workflow=SingleAgentVoiceWorkflow(agent),
    config=VoicePipelineConfig(tts_settings=custom_tts_settings),
)

Also optimize your agent’s own text output for speech: instruct it to write short, spoken-style sentences instead of markdown and bullets.

Voice Agents Still Use the Same Building Blocks

The voice surface changes the transport and audio loop, but everything else transfers directly:

• Tools — the voice agent calls the same function tools.
• Handoffs & orchestration — spoken workflows branch across specialists.
• Guardrails & human review — safety checks and approvals work unchanged.
• Sessions — conversation state persists across spoken turns.
• Observability — traces cover every model call, tool call, and handoff.

The practical rule: choose the audio architecture first, then design the rest of the agent workflow the same way you would for text.

Realtime Speech-to-Speech

When you need true conversational latency, use the realtime path. In JavaScript, that’s RealtimeAgent + RealtimeSession over WebRTC (browser) or WebSocket (server):

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a helpful voice assistant.",
});

const session = new RealtimeSession(agent, { model: "gpt-realtime-2.1" });
await session.connect({ apiKey: "ek_..." });  // ephemeral client token

Your server mints short-lived ephemeral tokens (POST /v1/realtime/client_secrets); the browser connects with WebRTC; the agent handles audio turns, tools, interruptions, and handoffs inside that session. In Python, the simplest path to extending a text agent into voice remains the chained VoicePipeline.

Putting It All Together

For a complete, runnable voice assistant—microphone capture, agent workflow, streaming TTS playback, and personality tuning—see the official cookbook and example:

• https://developers.openai.com/cookbook/examples/agents_sdk/app_assistant_voice_agents
• https://github.com/openai/openai-agents-python/tree/main/examples/voice/static

Conclusion & Next Steps

You now know how to turn a text agent into a voice assistant: a chained STT→agent→TTS VoicePipeline, streamed microphone input with activity detection, and TTS settings that shape the voice.

Next steps:

• Wrap your existing support agent in a VoicePipeline and add handoffs across specialists.
• Add guardrails and approval gates to the spoken workflow.
• Tune TTSModelSettings per use case and test with real recordings.
• Evaluate realtime speech-to-speech (WebRTC) when latency matters more than stage control.

Voice is the fastest-growing interface for agents. With a pipeline wrapper, the agents you’ve already built are minutes away from speaking.

References

• Voice agents quickstart
• Voice pipeline reference
• Voice agents guide
• Building a voice assistant with the Agents SDK