Multi-Turn Evals: Scoring Long Conversations End-to-End

Single-turn evals are straightforward: give the model a prompt, check the output. But real agent conversations span dozens of turns with branching decisions, tool calls, and evolving context. Multi-turn evals measure whether your agent maintains coherence, remembers critical information, and reaches correct conclusions across an entire session.

Why Single-Turn Evals Fall Short

A chatbot might score 95% on individual Q&A benchmarks but still fail catastrophically in production because:

• It contradicts itself after turn 10
• It forgets constraints established in turn 2
• It repeats the same tool call that already failed
• It loses track of the user’s goal when the conversation branches

Multi-turn evals capture these failure modes by scoring the entire trajectory, not isolated snapshots.

Anatomy of a Multi-Turn Eval

The Golden Trajectory

A golden trajectory is a reference conversation that represents ideal behavior:

golden_trajectory = {
    "task": "Book a flight from NYC to London for next Tuesday under $800",
    "turns": [
        {"role": "user", "content": "I need to fly from NYC to London next Tuesday"},
        {"role": "assistant", "content": "Let me search for flights...", "tool_calls": [{"name": "search_flights", "args": {"origin": "NYC", "destination": "LHR", "date": "2026-10-14"}}]},
        {"role": "tool", "content": "[3 results: $650, $720, $890]"},
        {"role": "assistant", "content": "Found 3 options. The $650 and $720 options are within your budget. The $650 flight has a 2-hour layover in Dublin."},
        {"role": "user", "content": "I prefer nonstop if possible"},
        {"role": "assistant", "content": "Let me filter for nonstop flights...", "tool_calls": [{"name": "search_flights", "args": {"origin": "NYC", "destination": "LHR", "date": "2026-10-14", "nonstop": True}}]},
        {"role": "assistant", "content": "The only nonstop option is $890, which exceeds your $800 budget. The best within budget is the $720 flight with a 1-hour layover. Shall I book that?"}
    ],
    "expected_outcome": "Recommended the $720 non-exact-match flight with explanation"
}

Scoring Dimensions

Multi-turn evals require scoring across multiple dimensions simultaneously:

from dataclasses import dataclass
from typing import Optional

@dataclass
class TurnScore:
    relevance: float       # Does this turn advance the task?
    coherence: float       # Is it consistent with previous turns?
    tool_accuracy: float   # Were the right tools called with correct args?
    memory: float          # Does it recall earlier context?

@dataclass
class TrajectoryScore:
    task_completion: float       # Did the agent finish the task?
    turn_scores: list[TurnScore]
    total_turns: int
    efficiency: float            # Did it minimize unnecessary turns?
    overall: float               # Weighted composite

Implementation with LangGraph

LangGraph’s checkpoint system makes it natural to evaluate multi-turn flows:

from langgraph.graph import StateGraph, MessagesState
from langgraph.checkpoint.memory import MemorySaver
from langsmith import evaluate, Client

# Define your agent graph
def agent_node(state: MessagesState):
    # Your agent logic here
    response = call_llm(state["messages"])
    return {"messages": [response]}

def tool_node(state: MessagesState):
    # Execute tool calls
    results = execute_tools(state["messages"])
    return {"messages": results}

graph = StateGraph(MessagesState)
graph.add_node("agent", agent_node)
graph.add_node("tools", tool_node)
graph.add_conditional_edges("agent", should_use_tools, {"yes": "tools", "no": "__end__"})
graph.add_edge("tools", "agent")

checkpointer = MemorySaver()
app = graph.compile(checkpointer=checkpointer)

# Define multi-turn eval target
def eval_target(inputs: dict) -> dict:
    config = {"configurable": {"thread_id": inputs["session_id"]}}
    for turn in inputs["conversation"]:
        result = app.invoke({"messages": [{"role": "user", "content": turn}]}, config)
    return {"messages": result["messages"]}

# Custom evaluator for trajectory quality
def trajectory_scorer(run, example):
    messages = run.outputs["messages"]
    golden = example.outputs["expected_trajectory"]
    
    # Score task completion
    task_complete = check_task_completion(messages, golden["task"])
    
    # Score context retention
    context_score = check_context_retention(messages, golden["constraints"])
    
    # Score tool usage efficiency
    tool_score = check_tool_efficiency(messages, golden["expected_tools"])
    
    return {
        "key": "trajectory_score",
        "score": (task_complete + context_score + tool_score) / 3,
        "comment": f"Task: {task_complete}, Context: {context_score}, Tools: {tool_score}"
    }

# Run evaluation
results = evaluate(
    eval_target,
    data="multi-turn-test-set",
    evaluators=[trajectory_scorer],
    experiment_prefix="v2-agent-eval"
)

Key Metrics for Multi-Turn Evals

Context Retention Score

Did the agent remember constraints from early turns?

def context_retention_score(messages: list, constraints: list[str]) -> float:
    scores = []
    for constraint in constraints:
        # Check if constraint is referenced in later turns
        mentioned = any(
            constraint.lower() in msg.content.lower()
            for msg in messages[3:]  # Skip first 3 turns
            if hasattr(msg, 'content')
        )
        scores.append(1.0 if mentioned else 0.0)
    return sum(scores) / len(scores) if scores else 0.0

Turn Efficiency Ratio

Did the agent reach the conclusion without wasting turns?

def efficiency_ratio(actual_turns: int, optimal_turns: int) -> float:
    if actual_turns <= optimal_turns:
        return 1.0
    # Penalize excess turns logarithmically
    import math
    excess = actual_turns - optimal_turns
    return max(0, 1.0 - math.log2(excess + 1) / 10)

Coherence Across Branches

When the user changes direction, does the agent adapt without losing track?

def coherence_score(messages: list) -> float:
    contradictions = 0
    for i, msg in enumerate(messages):
        if msg.role == "assistant":
            for j, earlier in enumerate(messages[:i]):
                if earlier.role == "assistant":
                    if contains_contradiction(earlier.content, msg.content):
                        contradictions += 1
    total_assistant_turns = sum(1 for m in messages if m.role == "assistant")
    return max(0, 1.0 - (contradictions / max(total_assistant_turns, 1)))

Building a Multi-Turn Eval Dataset

Curate Diverse Scenarios

Your eval set should cover:

• Happy path: Straightforward task completion
• Constraint changes: User modifies requirements mid-conversation
• Tool failures: Agent must recover from bad tool responses
• Ambiguous queries: Agent must ask clarifying questions
• Multi-step tasks: Requires 5+ tool calls to complete

eval_dataset = [
    {
        "scenario": "constraint_change",
        "conversation": [
            "I want a red sedan under $30K",
            "Actually, make it an SUV",
            "Does it have leather seats?"
        ],
        "expected_behavior": "Adapt to SUV, maintain price constraint, check features"
    },
    {
        "scenario": "tool_failure_recovery",
        "conversation": [
            "What's the weather in Paris?",
            "The API returned an error, try again",
            "Never mind, just tell me the forecast for tomorrow"
        ],
        "expected_behavior": "Retry once, then offer alternative approach"
    }
]

Production Evaluation Pipeline

# GitHub Actions integration
name: Agent Evals
on:
  pull_request:
    paths:
      - 'src/agent/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run multi-turn evals
        run: |
          python -m eval.run_multi_turn \
            --dataset golden-trajectories.json \
            --threshold 0.85 \
            --fail-on-regression
      - name: Comment PR with results
        uses: actions/github-script@v7
        with:
          script: |
            const results = require('./eval/results.json');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: `## Agent Eval Results\n- Trajectory Score: ${results.avg_score}\n- Context Retention: ${results.context_score}\n- Task Completion: ${results.task_score}`
            });

Common Pitfalls

1. Over-fitting to golden trajectories: Agent might take a different but equally valid path. Score outcomes, not exact steps.
2. Ignoring cost: A 3-turn solution at $0.01/turn beats a 10-turn solution at $0.005/turn.
3. Static eval sets: Update your eval data as your agent evolves. Stale evals give false confidence.
4. Missing edge cases: Always include conversations where the agent should refuse or escalate.

Multi-turn evals are computationally expensive but irreplaceable for production agents. Start with 50-100 curated trajectories covering your critical paths, and expand from there.