Evals for Agents: Unit Testing Multi-Step Reasoning

A function returns the same answer for the same input — an agent often doesn’t. It picks tools, makes intermediate decisions, and can succeed or fail in a dozen different ways. “Does it look right?” doesn’t cut it when each run costs tokens and takes minutes.

In this post, you will learn how to turn an agent’s run into something testable: split it into checkpoints, assert on tool calls and state transitions, and score the final answer with a rubric LLM. This is unit testing applied to multi-step reasoning.

Why you can’t snapshot-test an agent

Snapshot tests work when output is deterministic. Agents are stochastic: the same query may take a different number of steps, call tools in a different order, or even recover from a failure the first time and not the second. What’s stable is the shape of a good run:

• the agent called the right tool with the right arguments,
• it didn’t loop or stall,
• it reached a terminal state,
• the final answer satisfies the request.

Evals assert on those properties, not on exact text.

Instrument your agent with a trace

Before you can eval, you must observe. Wrap your agent loop so every step emits structured events — the tool called, its input/output, and the decision that followed:

from dataclasses import dataclass, field

@dataclass
class Step:
    kind: str                     # "thought" | "tool_call" | "tool_result"
    name: str | None = None       # tool name, e.g. "search_docs"
    input: dict = field(default_factory=dict)
    output: str | None = None

@dataclass
class AgentTrace:
    steps: list[Step] = field(default_factory=list)
    final_answer: str | None = None

def run_traced(agent, query: str) -> AgentTrace:
    trace = AgentTrace()
    for event in agent.run(query):            # your agent already streams events
        if event.type == "tool_call":
            trace.steps.append(Step(kind="tool_call", name=event.tool,
                                    input=event.arguments))
        elif event.type == "tool_result":
            trace.steps.append(Step(kind="tool_result", name=event.tool,
                                    output=event.result))
    trace.final_answer = agent.last_output
    return trace

Assert on the trace with plain pytest

The workhorse of agent evals is ordinary assert statements over the trace. Write them like you’d write any test:

import pytest

def test_uses_search_before_summarizing(make_agent):
    agent = make_agent()
    trace = run_traced(agent, "Summarize the latest release notes")
    calls = [s.name for s in trace.steps if s.kind == "tool_call"]
    assert "search_docs" in calls, f"expected search, got {calls}"
    assert calls.index("search_docs") < calls.index("summarize")

def test_terminates_without_looping(make_agent):
    agent = make_agent()
    trace = run_traced(agent, "List our top 5 customers")
    tool_calls = [s for s in trace.steps if s.kind == "tool_call"]
    assert len(tool_calls) <= 10, "agent looped"
    assert trace.final_answer, "agent returned no answer"

Score free-text answers with a rubric

Final answers are prose, so assert properties: contains the cited source, stays on topic, and is grounded. A judge LLM with an explicit rubric is the standard approach — the Hugging Face evaluation ecosystem and judgment style prompts make this easy:

from pydantic import BaseModel

class RubricResult(BaseModel):
    grounded: bool
    complete: bool
    score: float          # 0.0 - 1.0

RUBRIC = """You are grading an agent's answer to: {query}
Context the agent was given: {context}
Answer: {answer}

Score 0-1 on: (a) grounded in the context, (b) answers the full query,
(c) cites a source. Return JSON."""

def grade_answer(trace: AgentTrace, query: str, context: str) -> RubricResult:
    answer = trace.final_answer or ""
    return judge_llm.structured(RUBRIC, RubricResult,
                                query=query, context=context, answer=answer)

Run it over a fixed eval set (20–50 queries) and aggregate: pass if score >= 0.8. This is your regression baseline.

Wire evals into CI as a gate

An eval that never runs is a vibe, not a test. Add a job that executes the suite on every PR and blocks merges when the pass rate drops below a threshold:

# .github/workflows/agent-evals.yml
name: agent-evals
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .[evals]
      - run: python -m pytest tests/agents/ -m agent_eval
      - name: Check pass rate
        run: python scripts/check_eval_gate.py --min-pass-rate 0.9

The gate script reads the aggregated scores and exits non-zero if the suite regresses — so a prompt change that silently breaks retrieval now fails the build.

Putting It All Together

A complete template — traced agent harness, checkpoint tests, rubric grader, and the CI gate — is in this gist. Start with 5 happy-path queries and 5 edge cases; you’ll catch regressions on every subsequent change.

Conclusion & Next Steps

You can now test multi-step reasoning like code: checkpoint assertions for tool calls and loops, rubric scoring for answer quality, and a CI gate for regression protection. Next: add fuzz cases (adversarial inputs), track cost and latency per eval so a “better” prompt that triples tokens shows up, and version your eval set alongside the prompt it validates.

References / Sources

• Hugging Face evaluation tooling. https://huggingface.co/docs
• LlamaIndex evaluation module guides. https://docs.llamaindex.ai/en/stable/module_guides/evaluating/
• GitHub Actions docs. https://docs.github.com/en/actions