Handling Barge-In: Turn-Taking and Interruption for Voice Agents

The user starts talking while the AI is still explaining the refund policy. Without barge-in, they wait through the entire response. With barge-in done wrong, the AI stops every time the user clears their throat. The goal is natural conversation — and that’s a surprisingly hard engineering problem.

What Is Barge-In?

Barge-in is the ability of a voice AI to detect when a user starts speaking while the AI is still talking, then gracefully stop and process the new input. The technical challenge: distinguishing the user’s voice from the AI’s own audio output being picked up by the microphone.

Performance benchmarks (2026):

• Barge-in success rate: target >96%
• False barge-in rate: target <2%
• TTS flush latency: target <60ms
• Minimum interruption duration: 200-300ms with classifier confidence >0.7

The Layered Detection Stack

Production systems chain multiple layers because each one fails in different ways:

Audio Input
  └─→ Acoustic Denoising
       └─→ Energy VAD (speech vs. silence)
            └─→ Semantic VAD (thought complete vs. still speaking)
                 └─→ App-Level Interruption Gating

Energy VAD can’t tell a thinking-pause from a finished thought. Semantic classifiers calibrated on English mis-score Chinese clause pauses. Each layer covers the one below.

Voice Activity Detection (VAD) Architectures

Pattern	Latency	False Positive Rate	Complexity
Server VAD only	100-300ms	Medium	Low
Client VAD only	50-100ms	High	Medium
Hybrid	50-100ms	Low (2.4%)	High

The hybrid approach: local fast VAD fires immediately and pauses TTS (~50-100ms), server’s semantic VAD evaluates the audio (~200ms). If confirmed, TTS stays cancelled. If false positive, TTS resumes. This reduced false-positive rate from 11% to 2.4% in production.

Semantic VAD: When Is a Thought Complete?

OpenAI’s semantic_vad scores the probability the user completed their thought, governed by an eagerness parameter:

• low — 8s max wait
• medium/auto — 4s
• high — 2s

The minimum interruption duration is the single most useful control — audio on the caller channel doesn’t count unless it lasts long enough to be a word. A cough, door sound, or syllable from someone else fails this test at 300ms.

The Four-Step Barge-In Sequence

1. Detect overlap — VAD + semantic classification
2. Cancel server generation — In <40ms
3. Flush client playback — In <60ms
4. Reconcile server state — The step teams skip

State reconciliation is critical: when the user interrupts, the model has usually streamed the complete response and recorded the full transcript. Without correction, future turns are conditioned on words never spoken.

Gemini Live API Implementation

The Gemini Live API replaces the brittle STT→LLM→TTS pipeline with a single WebSocket session:

# Server VAD sends interruption signal
# When user speaks during model response:
{
    "serverContent": {
        "interrupted": True
    }
}

# Client must:
# 1. Stop playback immediately
# 2. Clear queued audio buffer
# 3. Discard buffered playback

Hybrid VAD with Gemini

1. Auto VAD remains enabled on server for speech onset detection
2. Client uses client-side VAD for end-of-speech detection
3. Client sends audio_stream_end signal
4. Server treats it as immediate finalization
5. Server-side VAD acts as fallback if client VAD fails

Known issues:

• 1011 errors on first user turn after greeting
• Model freezes after interrupted event — workaround: 4-second nudge timer
• No interrupted event when user is already speaking as model’s turn begins

Interruption Classification

Not all interruptions mean the same thing:

Type	Signal	Response
Correction	”No, use my work email”	Accept change, update value
Backchannel	”uh-huh”, “mm-hmm”	Keep speaking
Impatient	”Skip ahead”	Drop explanation, move on
Topic switch	Unrelated question	Answer briefly, steer back
Pushback	Challenge or distrust	De-escalate, explain

Backchannel detection rules:

1. Standalone affirmative token → Hold, keep speaking
2. Any negative token → Yield immediately
3. Affirmative token with continued speech → Yield, caller taking floor

Production Metrics

Track these to measure your barge-in quality:

• Interruption detection latency (speech start → playback stop)
• False interruption rate (backchannels stopping agent)
• Rollback correctness (partial output ends in right state)
• Correction success rate (next answer reflects changed intent)
• Repeated user speech rate (caller repeats same correction)

The Takeaway

Barge-in is a solved problem in the abstract but a hard one in production. Start with hybrid VAD, set a 300ms minimum interruption duration, and always implement state reconciliation after interruption. The Gemini Live API handles most of this server-side — your job is to flush playback and reconcile state on the client.

💡 Use audio_stream_end with the Gemini Live API for low-latency response finalization instead of waiting for silence detection.