Evaluating Voice Agents: WER, Latency, and Dialogue Quality Metrics That Matter

Building a voice agent is only half the battle. Evaluating it reliably is what separates a demo from a production system. Voice agents introduce unique evaluation challenges: speech recognition errors compound with LLM hallucinations, latency is measured in milliseconds, and conversation quality is subjective. This guide covers the metrics that actually matter.

The Evaluation Stack for Voice Agents

Voice agent evaluation operates across four layers:

┌─────────────────────────────────────┐
│        Dialogue Quality             │  ← User satisfaction, task completion
├─────────────────────────────────────┤
│        Response Latency             │  ← Time to first token, end-to-end
├─────────────────────────────────────┤
│        Tool Call Accuracy           │  ← Correct tool, correct args
├─────────────────────────────────────┤
│        Speech Recognition (WER)     │  ← Input understanding
└─────────────────────────────────────┘

Each layer depends on the one below it. A 99% WER is useless if latency is 10 seconds. Low latency means nothing if the agent calls the wrong tool.

Metric 1: Word Error Rate (WER)

WER measures how accurately the system transcribes user speech. It is the foundation—every downstream decision depends on correct transcription.

Formula

WER = (Substitutions + Insertions + Deletions) / Reference Word Count

Measuring WER

import evaluate

wer_metric = evaluate.load("wer")

references = [
    "What is the weather in Tokyo",
    "How much does the widget cost",
    "Transfer me to customer support"
]

predictions = [
    "What is the weather in Tokyo",      # Perfect
    "How much does the widget cost",     # Perfect
    "Transfer me to customer supports"   # Extra 's'
]

wer_score = wer_metric.compute(
    references=references,
    predictions=predictions
)
print(f"WER: {wer_score:.2%}")  # WER: 3.70%

WER Benchmarks for Voice Agents

WER Range	Quality	Use Case
0-5%	Excellent	Controlled environments, known speakers
5-10%	Good	Production voice agents with good audio
10-20%	Acceptable	Noisy environments, diverse accents
20%+	Poor	Needs acoustic model improvement

Factors That Affect WER

1. Audio quality: Background noise, microphone distance, codec compression
2. Speaker variation: Accents, speaking rate, pronunciation
3. Vocabulary: Domain-specific terms, proper nouns, technical jargon
4. Turn detection: Cutting off speech too early or too late

Metric 2: Response Latency

Latency is the time between when the user stops speaking and when the agent starts responding. For voice agents, this is the most critical user experience metric.

Latency Components

User stops speaking
    → VAD detection (50-200ms)
    → Audio transmission (50-100ms)
    → STT processing (200-500ms)
    → LLM inference (300-2000ms)
    → Tool execution (0-5000ms, if applicable)
    → TTS generation (200-800ms)
    → Audio playback (50-100ms)
    Agent starts speaking

Measuring End-to-End Latency

import time
import asyncio
from dataclasses import dataclass

@dataclass
class LatencyMeasurement:
    stt_latency: float      # Audio to text
    llm_latency: float      # Text to response
    tts_latency: float      # Response to audio
    tool_latency: float     # Tool execution
    total_latency: float    # End-to-end

class LatencyTracker:
    def __init__(self):
        self.measurements = []

    def measure_turn(self, audio_end_time: float, 
                     stt_end: float, llm_end: float,
                     tts_end: float, tool_time: float = 0) -> LatencyMeasurement:
        stt = stt_end - audio_end_time
        llm = llm_end - stt_end
        tts = tts_end - llm_end
        total = tts_end - audio_end_time + tool_time
        
        measurement = LatencyMeasurement(
            stt_latency=stt,
            llm_latency=llm,
            tts_latency=tts,
            tool_latency=tool_time,
            total_latency=total
        )
        self.measurements.append(measurement)
        return measurement

    def summary(self) -> dict:
        if not self.measurements:
            return {}
        
        totals = [m.total_latency for m in self.measurements]
        return {
            "mean": sum(totals) / len(totals),
            "p50": sorted(totals)[len(totals) // 2],
            "p95": sorted(totals)[int(len(totals) * 0.95)],
            "p99": sorted(totals)[int(len(totals) * 0.99)],
            "max": max(totals),
        }

Latency Budget

For a natural conversational feel, aim for these targets:

Component	Target	Acceptable
VAD + transmission	<150ms	<250ms
STT	<400ms	<700ms
LLM (without tools)	<500ms	<1000ms
LLM (with tools)	<2000ms	<3000ms
TTS	<400ms	<800ms
Total (no tools)	<1000ms	<2000ms
Total (with tools)	<3000ms	<5000ms

Reducing Latency

# Strategy 1: Use streaming TTS for faster first audio
# Start TTS before LLM completes
async def streaming_response(audio_input):
    # STT
    transcription = await stt_model.transcribe(audio_input)
    
    # Stream LLM response
    async for chunk in agent.stream(transcription):
        # Start TTS on first chunk
        tts_queue.put(chunk)
    
    # TTS consumes from queue in parallel
    audio = await tts_model.stream_from_queue(tts_queue)
    return audio

# Strategy 2: Pre-warm connections
# Keep model connections alive between requests
warm_client = OpenAIClient(keep_alive=True)

# Strategy 3: Use smaller models for simple queries
def route_by_complexity(query):
    if is_simple_query(query):
        return "gemini-flash"  # 200ms response
    else:
        return "gemini-pro"    # 800ms response

Metric 3: Tool Call Accuracy

Voice agents must correctly identify when to call tools and with what arguments. Errors here are often more costly than WER errors because they lead to wrong actions.

Measuring Tool Call Accuracy

from dataclasses import dataclass
from typing import Optional

@dataclass
class ToolCallEval:
    expected_tool: str
    expected_args: dict
    actual_tool: Optional[str]
    actual_args: Optional[dict]
    
    @property
    def tool_correct(self) -> bool:
        return self.expected_tool == self.actual_tool
    
    @property
    def args_correct(self) -> bool:
        if not self.actual_args:
            return False
        return self.expected_args == self.actual_args
    
    @property
    def fully_correct(self) -> bool:
        return self.tool_correct and self.args_correct

def evaluate_tool_calls(test_cases: list[ToolCallEval]) -> dict:
    total = len(test_cases)
    tool_correct = sum(1 for tc in test_cases if tc.tool_correct)
    args_correct = sum(1 for tc in test_cases if tc.args_correct)
    fully_correct = sum(1 for tc in test_cases if tc.fully_correct)
    
    return {
        "tool_accuracy": tool_correct / total,
        "argument_accuracy": args_correct / total,
        "overall_accuracy": fully_correct / total,
        "total_cases": total,
    }

# Example evaluation
test_cases = [
    ToolCallEval(
        expected_tool="get_weather",
        expected_args={"city": "Tokyo"},
        actual_tool="get_weather",
        actual_args={"city": "Tokyo"},
    ),
    ToolCallEval(
        expected_tool="check_inventory",
        expected_args={"product": "widget"},
        actual_tool="check_inventory",
        actual_args={"product": "widgets"},  # Pluralized
    ),
]

results = evaluate_tool_calls(test_cases)
# {"tool_accuracy": 1.0, "argument_accuracy": 0.5, "overall_accuracy": 0.5}

Common Tool Call Failures

1. Wrong tool selected: Agent calls check_stock instead of check_inventory
2. Missing arguments: Agent forgets required parameters
3. Argument hallucination: Agent invents values not in the user’s request
4. Unnecessary tool calls: Agent calls tools when the answer is in context
5. Failed to call tool: Agent answers from memory instead of checking

Metric 4: Dialogue Quality

Dialogue quality is subjective but can be measured through structured evaluation frameworks.

Task Completion Rate

Did the user accomplish what they set out to do?

def evaluate_task_completion(conversation: list[dict]) -> bool:
    """Evaluate if the task was completed successfully."""
    # Define success criteria per task type
    success_criteria = {
        "order_status": lambda conv: any(
            "shipped" in msg["content"].lower() or
            "delivered" in msg["content"].lower()
            for msg in conv if msg["role"] == "agent"
        ),
        "transfer": lambda conv: any(
            "transferring" in msg["content"].lower()
            for msg in conv if msg["role"] == "agent"
        ),
    }
    
    task_type = detect_task_type(conversation)
    if task_type in success_criteria:
        return success_criteria[task_type](conversation)
    
    return None  # Cannot determine

Conversation Naturalness Score

Rate how natural the conversation feels (1-5 scale):

naturalness_criteria = {
    5: "Indistinguishable from human conversation",
    4: "Natural with minor awkwardness",
    3: "Functional but noticeable AI patterns",
    2: "Frequent awkward turns, robotic responses",
    1: "Unusable for real conversation",
}

Multi-Turn Context Tracking

Does the agent remember previous turns?

def evaluate_context_tracking(conversation: list[dict]) -> dict:
    scores = {
        "pronoun_resolution": 0,    # "it" refers to correct entity
        "reference_tracking": 0,     # "that order" is understood
        "topic_persistence": 0,      # Stays on topic
        "correction_handling": 0,    # Handles self-corrections
    }
    
    for i, turn in enumerate(conversation):
        if turn["role"] == "user" and i > 0:
            # Check if agent correctly resolved references
            prev_context = [t for t in conversation[:i] if t["role"] == "user"]
            if any(pronoun in turn["content"] for pronoun in ["it", "that", "this"]):
                # Evaluate if the agent's response shows correct resolution
                pass  # Implementation depends on specific test cases
    
    return scores

Building an Evaluation Pipeline

Automated Evaluation Suite

import asyncio
from dataclasses import dataclass

@dataclass
class VoiceAgentEvalSuite:
    wer_threshold: float = 0.10       # 10% WER max
    latency_p95_threshold: float = 2.0  # 2 seconds max
    tool_accuracy_threshold: float = 0.90  # 90% min
    task_completion_threshold: float = 0.85  # 85% min
    
    async def run_evaluation(self, agent, test_cases):
        results = {
            "wer": await self.evaluate_wer(agent, test_cases),
            "latency": await self.evaluate_latency(agent, test_cases),
            "tool_calls": await self.evaluate_tool_calls(agent, test_cases),
            "dialogue": await self.evaluate_dialogue(agent, test_cases),
        }
        
        # Overall pass/fail
        results["passed"] = (
            results["wer"]["score"] <= self.wer_threshold and
            results["latency"]["p95"] <= self.latency_p95_threshold and
            results["tool_calls"]["accuracy"] >= self.tool_accuracy_threshold and
            results["dialogue"]["task_completion"] >= self.task_completion_threshold
        )
        
        return results

# Run evaluation
suite = VoiceAgentEvalSuite()
results = asyncio.run(suite.run_evaluation(agent, test_cases))

if results["passed"]:
    print("Agent ready for production")
else:
    print("Agent needs improvement")
    for metric, data in results.items():
        if metric != "passed" and not data.get("passed", True):
            print(f"  {metric}: {data}")

CI/CD Integration

# .github/workflows/voice-agent-eval.yml
name: Voice Agent Evaluation
on:
  pull_request:
    paths:
      - 'agents/voice/**'
      - 'tools/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run voice agent evals
        run: |
          python -m pytest tests/voice_eval/ \
            --wer-threshold=0.10 \
            --latency-threshold=2.0 \
            --tool-accuracy=0.90 \
            --tb=short

Key Takeaways

1. WER is your foundation: If transcription fails, everything downstream fails
2. Latency is UX: Users abandon voice agents that take more than 2 seconds
3. Tool accuracy matters more than chat accuracy: Wrong tools mean wrong actions
4. Evaluate in context: Isolated metrics miss multi-turn conversation quality
5. Automate everything: Manual evaluation does not scale

Voice agent evaluation is an ongoing process. As your agent handles more real conversations, update your test suite with edge cases from production. The best evaluation pipelines combine automated metrics with periodic human review.

References:

• Hugging Face Evaluate Library
• OpenAI Agents SDK Testing
• WER Benchmarks for Speech Recognition