Realtime Voice Agents with Google ADK: Building Low-Latency Conversational AI

Voice agents are no longer a novelty. In 2026, they are a production requirement. With Google’s Agent Development Kit (ADK), building low-latency, tool-calling voice agents is faster than ever. This guide walks you through the full pipeline—from setting up a streaming voice agent to handling interruptions and tool calls mid-conversation.

Why Voice Agents Need a Different Architecture

Traditional text-based agents process inputs as discrete turns. Voice agents operate on continuous audio streams with strict latency requirements. A user expects a response within 300-500ms, not the 2-5 seconds typical of text-based LLM chains.

Google ADK solves this with its Live and Voice Agents system, which provides:

• Bidirectional streaming audio via WebSocket connections
• Automatic interruption detection so users can barge in mid-sentence
• Tool calling during conversations without breaking the audio flow
• Session management that persists context across turns

Prerequisites

• Python 3.10+
• A Google Cloud project with the Gemini API enabled
• google-adk package installed
• A model that supports streaming (e.g., gemini-2.0-flash-live-001)

Setting Up Your First Voice Agent

Installation

pip install google-adk

Basic Voice Agent

The simplest voice agent uses ADK’s LlmAgent with a live-capable model:

from google.adk import Agent
from google.adk.tools import FunctionTool

# Define a tool the agent can call mid-conversation
def get_weather(city: str) -> dict:
    """Get current weather for a city."""
    return {
        "city": city,
        "temperature": "22C",
        "condition": "sunny"
    }

weather_tool = FunctionTool(func=get_weather)

# Create the agent
agent = Agent(
    name="weather_assistant",
    model="gemini-2.0-flash-live-001",
    instruction="""You are a friendly weather assistant.
    You help users check weather conditions.
    Keep responses short and conversational.""",
    tools=[weather_tool],
)

Running the Agent with ADK Web

ADK ships with a built-in browser client that captures microphone input, plays responses, and renders the transcript. No client code required:

adk web

This opens a local web interface where you can immediately talk to your agent. The browser client handles:

• Microphone capture and audio encoding
• WebSocket connection management
• Audio playback of agent responses
• Transcript rendering

Building a Custom Voice Pipeline

For production deployments, you will want more control over the audio pipeline. ADK supports custom servers with the VoicePipeline abstraction.

Architecture Overview

Audio Input → STT Model → Agent (with tools) → TTS Model → Audio Output

The pipeline chains three components:

1. Speech-to-Text (STT): Converts raw audio to text transcriptions
2. Agent Workflow: Processes the transcription and generates responses
3. Text-to-Speech (TTS): Converts the agent’s text response back to audio

Custom Server Implementation

from google.adk.agents import LlmAgent
from google.adk.tools import FunctionTool

# Define tools for your voice agent
def lookup_order(order_id: str) -> str:
    """Look up order status by ID."""
    return f"Order {order_id} is shipping tomorrow."

def transfer_to_human(reason: str) -> str:
    """Transfer the call to a human agent."""
    return f"Transferring to human agent. Reason: {reason}"

# Build the agent with multiple tools
agent = LlmAgent(
    name="support_agent",
    model="gemini-2.0-flash-live-001",
    instruction="""You are a customer support agent.
    Help users with order inquiries and transfers.
    Be concise and natural in conversation.""",
    tools=[
        FunctionTool(func=lookup_order),
        FunctionTool(func=transfer_to_human),
    ],
)

Handling Interruptions

Real conversations involve interruptions. Users will speak over the agent, change topics mid-sentence, or correct themselves. ADK’s live system handles this through VAD (Voice Activity Detection):

# Configuration for turn detection
config = {
    "voice_activity_detection": {
        "silence_duration_ms": 500,      # How long silence = end of turn
        "prefix_padding_ms": 300,         # Audio before speech detection
        "suffix_padding_ms": 200,         # Audio after speech stops
    }
}

When a user interrupts:

1. The current agent response is cancelled
2. The new user input is processed immediately
3. The agent resumes with updated context

This creates a natural conversational flow where users do not need to wait for the agent to finish speaking.

Tool Calling in Voice Conversations

The real power of voice agents is their ability to take actions. ADK tools are called mid-conversation without breaking the audio stream.

Example: Voice-Activated Order System

from google.adk import Agent
from google.adk.tools import FunctionTool

def check_inventory(product: str) -> dict:
    """Check if a product is in stock."""
    inventory = {
        "widget": {"in_stock": True, "quantity": 42},
        "gadget": {"in_stock": False, "quantity": 0},
    }
    product_lower = product.lower()
    if product_lower in inventory:
        return inventory[product_lower]
    return {"in_stock": False, "quantity": 0, "error": "Product not found"}

def place_order(product: str, quantity: int) -> dict:
    """Place an order for a product."""
    return {
        "status": "confirmed",
        "product": product,
        "quantity": quantity,
        "estimated_delivery": "2026-09-20"
    }

agent = Agent(
    name="shop_assistant",
    model="gemini-2.0-flash-live-001",
    instruction="""You are a helpful shopping assistant.
    Help users check inventory and place orders.
    Confirm details before placing orders.""",
    tools=[
        FunctionTool(func=check_inventory),
        FunctionTool(func=place_order),
    ],
)

When a user says “Do you have widgets in stock?”, the agent:

1. Transcribes the audio
2. Recognizes the tool call needed
3. Calls check_inventory("widget")
4. Generates a spoken response
5. Converts text to audio and plays it back

All within the latency budget of a natural conversation.

Sessions and Context Management

Voice conversations need persistent context. ADK sessions maintain conversation history across turns:

# Sessions persist across interactions
session_config = {
    "session_id": "user_123_session",
    "state": {
        "user_name": "Alice",
        "previous_orders": ["ORD-001", "ORD-002"]
    }
}

The agent can reference previous turns: “You mentioned earlier that you ordered widgets—would you like to check on that order?”

Evaluation and Testing

Voice agents require different evaluation than text agents. ADK provides built-in evaluation tools:

# Evaluate voice agent quality
from google.adk.evaluation import evaluate_voice_agent

results = evaluate_voice_agent(
    agent=agent,
    test_cases=[
        {
            "input": "What's the weather in Tokyo?",
            "expected_tool_call": "get_weather",
            "expected_args": {"city": "Tokyo"}
        },
        {
            "input": "Transfer me to a human",
            "expected_tool_call": "transfer_to_human"
        }
    ]
)

Key metrics for voice agents:

• Word Error Rate (WER): How accurately the agent understood speech
• Response Latency: Time from user speech end to agent response start
• Tool Call Accuracy: Whether the agent called the correct tool with right arguments
• Turn-Taking Quality: How well the agent handles interruptions and pauses

Production Deployment

Deploying to Google Cloud

ADK agents deploy natively to Google Cloud services:

# Deploy to Cloud Run
adk deploy cloud-run your_agent_directory

# Deploy to GKE
adk deploy gke your_agent_directory

Scaling Considerations

Factor	Recommendation
Concurrent sessions	Use connection pooling per instance
Audio codec	Prefer Opus for bandwidth efficiency
Model selection	Use Gemini Flash for low latency
Tool execution	Offload to async workers for long operations

Best Practices

1. Keep responses short: Voice users prefer 1-2 sentence answers
2. Handle barge-in gracefully: Cancel responses cleanly when interrupted
3. Use confirmations for critical actions: “Placing your order now—shall I proceed?”
4. Provide audio feedback: Subtle sounds for processing, errors, and completions
5. Test with real audio: Synthetic text tests miss pronunciation and accent issues

Conclusion

Google ADK makes building production voice agents accessible. The combination of streaming audio, tool calling, and session management gives you everything needed for conversational AI that feels natural. Start with adk web for rapid prototyping, then graduate to custom servers for production deployments.

The next frontier is multi-modal voice agents that combine audio with video and screen sharing—ADK is already positioned to support these use cases with its multimodal model support.

References:

• Google ADK Documentation
• ADK Live Agents Guide
• Supported Models for Voice