Skip to content
Blog

Voice Interfaces: Speech-to-Text to LLM to TTS Pipelines

Under 1 second of perceived latency for a voice agent. Break down the STT -> LLM -> TTS pipeline, learn when streaming beats speech-to-speech, and engineer the latency budget line by line.

Published on • August 8, 2026

AI Assistant

A voice agent that makes you wait a vocal pause feels like a phone tree. The gap between “feels instant” and “feels like a bot” is about 300–1000 ms of engineered latency — and almost all of it is orchestration, not model quality. This post breaks down the STT → LLM → TTS pipeline, where the milliseconds go, and when to skip the pipeline entirely.

The three-stage pipeline

A “cascaded” voice agent chains three specialized models:

graph LR
    Start[Microphone] --> VAD[VAD]
    VAD --> STT[STT]
    STT --> Text1[text]
    Text1 --> LLM[LLM]
    LLM --> Text2[text]
    Text2 --> TTS[TTS]
    TTS --> End[Speaker]

    style Text1 fill:#f9f,stroke:#333,stroke-width:2px,color:black
    style Text2 fill:#f9f,stroke:#333,stroke-width:2px,color:black
  • STT (speech-to-text): transcribes audio. Streaming STT emits partial transcripts while the user is still speaking.
  • LLM: the reasoning layer. It consumes partial transcript + context/stystem prompt.
  • TTS (text-to-speech): synthesizes audio from the model’s text. Streaming TTS starts speaking from the first sentence fragment.

The reason to cascade instead of one Big Model: control. In an entry you can mix and match providers (best STT, best LLM, best TTS), swap any stage independently, keep an exact transcript for compliance, and do function calling. The cost: latency per stage and more moving parts.

The naive (sequential) pipeline

Wait for full utterance → send whole audio to STT (150–500ms) → wait for full LLM response (350ms-1s+) → send full text to TTS (100–500ms) → play. That adds up to 600ms–2s before a word is heard. Fine for a prototype, not for production.

Streaming is the real optimization

The single biggest latency win is overlapping the stages: streaming STT emits partial transcripts, the LLM starts on the partial, TTS begins speaking on the first sentence. In practice, streaming saves 300–600ms end-to-end depending on the provider’s streaming support.

A realistic budget once everything streams:

StageTime budget
STT finalization50–100ms
LLM time-to-first-token100–200ms
TTS time-to-first-byte50–80ms
Transport (WebRTC)20–50ms
Total (target)300ms

The sentence buffer is the crucial orchestration primitive: it accumulates LLM tokens and only hands whole sentences to TTS, so the synth starts cleanly mid-stream instead of choking on partial words.

Picking models for voice is picking TTF-by-budget

For voice, the binding constraint is time-to-first-token, not benchmark score. Fast models land in the 50–300ms band (Groq-hosted, gemini flash-lite; gpt-4o-mini ~120–200ms), while frontier reasoning models are measured in seconds — they’re off the table for live voice. If you need capability + liveness, hedge: run two LLMs in parallel and use whichever returns a usable first token first.

Endpointing matters too: naive VAD waits ~600ms of silence before deciding the user finished — that alone blows a 400ms budget. Semantic turn detection (pitch + energy + lexical cues) brings end-of-turn under 300ms and cuts false interruptions ~45%.

When to skip it: speech-to-speech models

Gemini Live, OpenAI Realtime, Nova Sonic, and friends accept raw audio and return raw audio: one model, no text in between. Latency drops under ~500ms total and prosody is natural. The tradeoffs:

  • No transcript by default — you’ll want parallel STT for display, compliance, RAG.
  • Provider lock-in: the LLM and the voice are the same model; you can’t swap one without the other.
  • Function calling is less mature than in a text-based cascade.

Production reality (from the arXiv tutorial): native Qwen2.5-Omni reached ~13-second time-to-first-audio, whereas a streaming cascade (Deepgram + vLLM + ElevenLabs) delivered a measured 755ms with working function calls — ~17× faster. So “simplest to build” (speech-to-speech) is not yet “simplest to control”: choose realtime for raw speed in narrow tasks, cascade when you need control, and hybrid when you need both (S2S conversation + a parallel STT stream for transcripts/audit).

Putting It All Together

Whatever architecture, capture two streams on every call:

  1. the conversation audio, and
  2. a transcript stream from the same audio (even when the realtime model is speech-to-speech), because transcripts power eval, compliance, and your RAG retrieval.

Conclusion & Next Steps

Map your budget: pick an STT, an LLM, and a TTS that each meet their line item, then make every stage streaming. Add semantic endpointing instead of VAD-only silence detection, and wrap the whole loop in an eval harness that measures time-to-first-audio — not just overall quality — since that’s the number your users feel. Next: wire function calling into your LLM stage so the voice agent can act (book, order, transfer) without special-casing audio in your product layer.

References / Sources

ai voice stt tts pipelines