Agent Evals in CI: Suites That Run on Every Commit

Agent evals shouldn’t be a monthly audit—they should run on every commit. Integrating eval suites into CI catches regressions before they ship, tracks quality over time, and gives your team confidence to iterate.

The Challenge of Agent Evals in CI

Unlike unit tests, agent evals:

• Are expensive: Each eval run calls an LLM
• Are slow: Multi-turn conversations take seconds to minutes
• Are non-deterministic: Same input can produce different outputs
• Need thresholds: A single failure doesn’t mean regression

The solution: a tiered eval system with fast smoke tests on every commit and full evals on merge.

Architecture

┌─────────────┐     ┌─────────────────┐     ┌──────────────────┐
│  PR Created │────▶│  Fast Evals     │────▶│  Gate / No-Gate  │
│             │     │  (50 cases,     │     │                  │
│             │     │   < 2 min)      │     │                  │
└─────────────┘     └─────────────────┘     └──────────────────┘
                          │
                    ┌─────┴──────┐
                    │  Full Evals │
                    │  (500 cases,│
                    │   < 15 min) │
                    └────────────┘

Implementation with OpenAI Agents SDK

from agents import Agent, Runner
from dataclasses import dataclass
import json
import time

@dataclass
class EvalCase:
    id: str
    input: str
    expected_output: str | None = None
    expected_tool: str | None = None
    max_turns: int = 10
    tags: list[str] = None

@dataclass
class EvalResult:
    case_id: str
    passed: bool
    score: float
    latency_ms: float
    tool_calls: list[str]
    actual_output: str
    error: str | None = None

class AgentEvalSuite:
    def __init__(self, agent: Agent, cases: list[EvalCase]):
        self.agent = agent
        self.cases = cases
        self.results: list[EvalResult] = []
    
    def run(self, max_cases: int | None = None) -> list[EvalResult]:
        cases_to_run = self.cases[:max_cases] if max_cases else self.cases
        self.results = []
        
        for case in cases_to_run:
            result = self._eval_single(case)
            self.results.append(result)
        
        return self.results
    
    def _eval_single(self, case: EvalCase) -> EvalResult:
        start_time = time.time()
        
        try:
            result = Runner.run_sync(self.agent, case.input)
            latency_ms = (time.time() - start_time) * 1000
            
            # Score the result
            score = self._score_result(case, result)
            passed = score >= 0.7
            
            # Extract tool calls from trace
            tool_calls = self._extract_tool_calls(result)
            
            return EvalResult(
                case_id=case.id,
                passed=passed,
                score=score,
                latency_ms=latency_ms,
                tool_calls=tool_calls,
                actual_output=result.final_output,
            )
        except Exception as e:
            return EvalResult(
                case_id=case.id,
                passed=False,
                score=0.0,
                latency_ms=(time.time() - start_time) * 1000,
                tool_calls=[],
                actual_output="",
                error=str(e)
            )
    
    def _score_result(self, case: EvalCase, result) -> float:
        score = 0.0
        
        # Output similarity (if expected output provided)
        if case.expected_output:
            similarity = self._text_similarity(result.final_output, case.expected_output)
            score += similarity * 0.5
        
        # Tool selection (if expected tool provided)
        if case.expected_tool:
            tool_calls = self._extract_tool_calls(result)
            if case.expected_tool in tool_calls:
                score += 0.3
        
        # Basic quality checks
        if result.final_output and len(result.final_output) > 10:
            score += 0.2
        
        return min(1.0, score)
    
    def summary(self) -> dict:
        if not self.results:
            return {}
        
        return {
            "total": len(self.results),
            "passed": sum(1 for r in self.results if r.passed),
            "failed": sum(1 for r in self.results if not r.passed),
            "pass_rate": sum(1 for r in self.results if r.passed) / len(self.results),
            "avg_score": sum(r.score for r in self.results) / len(self.results),
            "avg_latency_ms": sum(r.latency_ms for r in self.results) / len(self.results),
            "errors": sum(1 for r in self.results if r.error),
        }

GitHub Actions Integration

# .github/workflows/agent-evals.yml
name: Agent Evals
on:
  pull_request:
    paths:
      - 'src/agent/**'
      - 'src/tools/**'
  push:
    branches: [main]

jobs:
  fast-evals:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run fast eval suite
        run: |
          python -m eval.run \
            --suite fast \
            --max-cases 50 \
            --threshold 0.80 \
            --output results.json
      
      - name: Check regression
        if: failure()
        run: |
          echo "## Agent Eval Regression" >> $GITHUB_STEP_SUMMARY
          echo "Pass rate dropped below 80% threshold" >> $GITHUB_STEP_SUMMARY
      
      - name: Comment PR
        uses: actions/github-script@v7
        with:
          script: |
            const results = require('./results.json');
            const emoji = results.pass_rate >= 0.80 ? '✅' : '❌';
            const body = `## ${emoji} Agent Eval Results
            
            | Metric | Value |
            |--------|-------|
            | Pass Rate | ${(results.pass_rate * 100).toFixed(1)}% |
            | Avg Score | ${results.avg_score.toFixed(3)} |
            | Avg Latency | ${results.avg_latency_ms.toFixed(0)}ms |
            | Failed Cases | ${results.failed}/${results.total} |
            
            [View full results](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})`;
            
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: body
            });

  full-evals:
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Run full eval suite
        run: |
          python -m eval.run \
            --suite full \
            --output results.json
      
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results-${{ github.sha }}
          path: results.json
      
      - name: Update eval dashboard
        run: |
          python -m eval.update_dashboard \
            --results results.json \
            --commit ${{ github.sha }}

Regression Detection

import json
from pathlib import Path

class RegressionDetector:
    def __init__(self, baseline_file: str):
        self.baseline = self._load_baseline(baseline_file)
    
    def _load_baseline(self, path: str) -> dict:
        if Path(path).exists():
            with open(path) as f:
                return json.load(f)
        return {"pass_rate": 1.0, "avg_score": 1.0}
    
    def check(self, current: dict, thresholds: dict | None = None) -> dict:
        thresholds = thresholds or {
            "pass_rate_drop": 0.05,    # Max 5% drop
            "score_drop": 0.03,         # Max 3% drop
            "new_failures": 5,           # Max 5 new failures
        }
        
        regressions = []
        
        # Check pass rate
        pass_rate_drop = self.baseline["pass_rate"] - current["pass_rate"]
        if pass_rate_drop > thresholds["pass_rate_drop"]:
            regressions.append({
                "metric": "pass_rate",
                "baseline": self.baseline["pass_rate"],
                "current": current["pass_rate"],
                "drop": pass_rate_drop,
                "threshold": thresholds["pass_rate_drop"]
            })
        
        # Check average score
        score_drop = self.baseline["avg_score"] - current["avg_score"]
        if score_drop > thresholds["score_drop"]:
            regressions.append({
                "metric": "avg_score",
                "baseline": self.baseline["avg_score"],
                "current": current["avg_score"],
                "drop": score_drop,
                "threshold": thresholds["score_drop"]
            })
        
        return {
            "regressed": len(regressions) > 0,
            "regressions": regressions,
            "baseline": self.baseline,
            "current": current
        }

Tiered Eval Strategy

EVAL_TIERS = {
    "smoke": {
        "description": "Critical path validation, runs on every commit",
        "max_cases": 10,
        "timeout_seconds": 60,
        "threshold": 0.90,
    },
    "fast": {
        "description": "Core functionality, runs on PRs",
        "max_cases": 50,
        "timeout_seconds": 120,
        "threshold": 0.80,
    },
    "full": {
        "description": "Comprehensive evaluation, runs on merge to main",
        "max_cases": 500,
        "timeout_seconds": 900,
        "threshold": 0.75,
    },
    "nightly": {
        "description": "Extended eval including edge cases",
        "max_cases": 2000,
        "timeout_seconds": 3600,
        "threshold": 0.70,
    }
}

Running agent evals on every commit requires investment in eval infrastructure, but it’s the only way to ship agents with confidence. Start with a smoke test suite of 10-20 critical cases and expand from there.