Building an Eval Test Suite for ADK Agents

Improving agent quality requires transitioning from ad-hoc manual testing (“vibes-based testing”) to structured, continuous evaluation. When you modify prompt instructions, register new tools, or upgrade model versions, an automated evaluation suite guarantees that performance improves without introducing silent regressions.

What Makes a Good Eval Suite

A robust agent evaluation suite must measure two distinct operational dimensions:

1. Final Output Quality: Does the agent produce correct, factual, and helpful text responses for the end user?
2. Intermediate Trajectory Accuracy: Did the agent select the correct sequence of tools and pass accurately parsed arguments?

Both dimensions are essential. A correct final response derived through an improper tool trajectory is a hidden failure—it will inevitably break when exposed to real-world edge cases.

Dataset Schema

Structure test cases by defining the input prompt alongside expected tool call trajectories and response assertions:

{
  "eval_cases": [
    {
      "id": "tc_billing_01",
      "input_prompt": "I need a refund for invoice INV-100.",
      "expected_tools": [
        {
          "name": "get_invoice_details",
          "args": {"invoice_id": "INV-100"}
        },
        {
          "name": "process_refund",
          "args": {"invoice_id": "INV-100"}
        }
      ],
      "expected_response_contains": ["refund", "processed", "INV-100"]
    }
  ]
}

Key Metrics to Track

Metric	What It Measures	How to Score
Tool Call Accuracy	Did the agent invoke the correct set of tools?	Exact match on expected tool names
Argument Extraction	Did the agent accurately extract parameter values from input?	Exact match or regex matching on argument values
Trajectory Efficiency	Did the agent avoid redundant or loop tool calls?	Count of actual tool calls vs. optimal baseline
Response Accuracy	Does the final text correctly answer the prompt?	Structured LLM-as-a-Judge evaluation

LLM-as-a-Judge Scoring

To score open-ended text quality, employ an independent LLM instance as an automated judge:

judge_prompt = """
Score the following agent response on a scale of 1 to 5 based on the provided ground truth:

1. Faithfulness: Is the response strictly grounded in the tool execution results?
2. Helpfulness: Does the response directly answer the user's initial prompt?

Agent Response: {response}
Ground Truth Reference: {expected}
"""

Common Judge Metrics

• Faithfulness (1–5): Ensures zero unverified claims or hallucinations beyond what the tools returned.
• Helpfulness (1–5): Evaluates whether the user’s intent was directly resolved.
• Safety & Compliance (Pass/Fail): Verifies adherence to policy guidelines, safety boundaries, and tone constraints.

Running Evals with agents-cli

Google ADK provides built-in support for execution and reporting via the agents-cli eval command:

# Run evaluation dataset against a specific agent project
agents-cli eval --dataset eval_dataset.json --agent my_agent_project

# Verbose mode with complete step-by-step trajectory trace
agents-cli eval --dataset eval_dataset.json --agent . --verbose

# Export structured JSON evaluation reports
agents-cli eval --dataset eval_dataset.json --output-file reports/eval_results.json

The CLI executes each test case asynchronously, verifies agent trajectories against ground truth references, and generates clear evaluation reports.

Continuous Evaluation in Development Workflow

Integrate evaluation directly into your daily development cycle:

1. Benchmark Before Changes: Run evaluation suites prior to modifying prompt instructions or tool signatures to establish baseline pass rates.
2. Enforce Quality Gates: Fail continuous integration builds if pass rates drop below predetermined thresholds.
3. Track Quality Trends: Store historical evaluation reports in data storage to detect gradual performance drift across model updates.

Key Takeaway

Automated eval suites uncover trajectory bugs—such as wrong tool selection or incorrect parameter formatting—that manual testing overlooks. Start by creating a compact evaluation dataset of 5–10 core user scenarios, and systematically expand it whenever edge-case bugs are identified.