Automated CI/CD Regression Evaluation for ADK Agents

Manual evaluation quickly becomes unsustainable as projects grow. When pull requests introduce prompt refinements, new tool definitions, or updated model parameters, you need automated regression evaluation in CI/CD pipelines to block deployments whenever quality metrics fall below strict thresholds.

This guide details how to build a Python evaluation pipeline that programmatically executes an ADK agent, validates trajectory events, and enforces an automated quality gate.

Pipeline Architecture

Git Push → CI Trigger → Run Unit Tests → Run Eval Pipeline
                                            ↓
                                    Pass Rate >= 100%?
                                   /                  \
                                YES                    NO
                                 ↓                      ↓
                           Deploy to              Block deployment,
                           Cloud Run              notify dev team

Complete Python Recipe: Automated Eval Pipeline

This standalone evaluation pipeline runs test prompts against an ADK agent instance, inspects session.events to verify tool calls and arguments, and enforces an exit code of 0 on success or 1 on failure:

import asyncio
import json
from google.adk.agents import Agent
from google.adk.tools import FunctionTool
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService

# 1. Target Function & Tool Definition
def lookup_order_status(order_id: str) -> dict:
    """Fetches shipping status for a given order ID.

    Args:
        order_id: The unique order code (e.g. 'ORD-55').
    """
    if order_id == "ORD-55":
        return {"status": "SHIPPED", "carrier": "FedEx", "tracking": "1Z999"}
    return {"status": "NOT_FOUND"}

order_tool = FunctionTool(func=lookup_order_status)

# 2. Agent Under Test
agent_under_test = Agent(
    name="order_agent",
    model="gemini-2.5-flash",
    instruction="Use `lookup_order_status` to answer order tracking questions.",
    tools=[order_tool]
)

# 3. Evaluation Test Suite Dataset
EVAL_DATASET = [
    {
        "id": "test_01",
        "prompt": "Where is my order ORD-55?",
        "expected_tool": "lookup_order_status",
        "expected_arg_key": "order_id",
        "expected_arg_value": "ORD-55"
    }
]

# 4. Asynchronous Evaluation Runner
async def run_eval_pipeline() -> bool:
    print("--- Starting Automated CI/CD Eval Pipeline ---")
    session_service = InMemorySessionService()
    runner = Runner(agent=agent_under_test, session_service=session_service)

    passed_count = 0
    total_count = len(EVAL_DATASET)

    for test_case in EVAL_DATASET:
        print(f"\n[RUNNING TEST] {test_case['id']}: '{test_case['prompt']}'")

        session_id = f"eval-sess-{test_case['id']}"
        res = await runner.run_async(session_id=session_id, message=test_case["prompt"])

        session = await session_service.get_session(session_id)

        # Trajectory inspection: Verify tool call and arguments in session events
        tool_called = False
        arg_correct = False

        for event in session.events:
            if hasattr(event, "tool_calls") and event.tool_calls:
                for call in event.tool_calls:
                    if call.get("name") == test_case["expected_tool"]:
                        tool_called = True
                        args = call.get("args", {})
                        if args.get(test_case["expected_arg_key"]) == test_case["expected_arg_value"]:
                            arg_correct = True

        if tool_called and arg_correct:
            print(f"[PASS] {test_case['id']} successfully invoked tool with correct arguments!")
            passed_count += 1
        else:
            print(f"[FAIL] {test_case['id']} failed trajectory validation.")

    pass_rate = (passed_count / total_count) * 100.0
    print(f"\n--- Eval Results: {passed_count}/{total_count} Passed ({pass_rate:.1f}%) ---")

    return pass_rate >= 100.0

async def main():
    success = await run_eval_pipeline()
    if not success:
        print("ERROR: CI Eval Quality Gate Failed! Blocking deployment.")
        exit(1)
    print("SUCCESS: Quality Gate Passed. Ready for production deployment!")

if __name__ == "__main__":
    asyncio.run(main())

Integrating into GitHub Actions

Add the evaluation pipeline step into your GitHub Actions workflow file (.github/workflows/deploy.yml) right before container image build commands:

- name: Run Test & Eval Quality Gate
  run: |
    uv sync
    uv run python -m pytest tests/
    uv run python recipes/eval_pipeline.py

If the pass rate drops below 100%, the step fails and aborts the workflow execution before container creation, preventing faulty prompt updates from deploying to cloud environments.

Scaling the Evaluation Suite

Start with a targeted suite of 5–10 core scenarios and progressively expand coverage:

• Edge Cases: Evaluate behavior on empty inputs, ambiguous queries, or invalid parameters.
• Regression Cases: Convert resolved production bugs into permanent evaluation test cases.
• Multi-Tool Selection: Include test cases that require selecting the single correct tool out of multiple similar tool definitions.

Key Takeaway

Enforcing a 100% trajectory accuracy gate within CI workflows catches silent errors where agents generate plausible responses via incorrect tool execution paths. It provides an automated, reliable safety barrier for every prompt edit and codebase change.