A/B Testing Agent Variants: Prompts, Models, and Context

A/B testing agents isn’t like A/B testing web pages. You can’t just swap a button color—your variants might use different models, different prompts, or different context strategies. LiteLLM’s gateway makes this practical by routing traffic, tracking costs, and collecting metrics across variants without changing your application code.

Why Agent A/B Testing Is Hard

Traditional A/B testing assumes deterministic outputs. Agent A/B testing faces unique challenges:

• Non-deterministic: Same prompt can produce different results
• Multi-turn: Variants affect entire conversation trajectories
• Cost asymmetry: GPT-4o costs 20x more than GPT-4o-mini
• Latency variance: Different models have different response times
• Tool-calling differences: Variants may choose different tools

Architecture with LiteLLM

┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Client     │────▶│  LiteLLM Proxy   │────▶│  Model A (GPT4o)│
│              │     │                  │     └─────────────────┘
│              │     │  ┌────────────┐  │     ┌─────────────────┐
│              │────▶│  │  Router    │────▶│  Model B (Claude) │
│              │     │  │  + Logger  │  │     └─────────────────┘
│              │     │  └────────────┘  │
└─────────────┘     └──────────────────┘

Setting Up the Gateway

# litellm_config.yaml
model_list:
  - model_name: "variant-a"
    litellm_params:
      model: "openai/gpt-4o"
      api_key: os.environ/OPENAI_API_KEY
  - model_name: "variant-b"
    litellm_params:
      model: "anthropic/claude-sonnet-4-20250514"
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: "variant-c"
    litellm_params:
      model: "openai/gpt-4o-mini"
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: "simple-shuffle"  # Or custom AB test routing
  num_retries: 3
  timeout: 30

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Custom A/B Routing

import hashlib
import json
from fastapi import FastAPI, Request
from litellm.proxy.proxy_server import app as litellm_app

# Define variants
VARIANTS = {
    "prompt_v1": {"model": "variant-a", "system_prompt": "You are a helpful assistant."},
    "prompt_v2": {"model": "variant-a", "system_prompt": "You are a concise, expert assistant. Answer in under 100 words."},
    "model_swap": {"model": "variant-b", "system_prompt": "You are a helpful assistant."},
    "cheap_model": {"model": "variant-c", "system_prompt": "You are a helpful assistant."},
}

def assign_variant(user_id: str, experiment: str) -> str:
    """Deterministic variant assignment using consistent hashing."""
    hash_input = f"{user_id}:{experiment}"
    hash_val = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
    variant_names = list(VARIANTS.keys())
    return variant_names[hash_val % len(variant_names)]

Tracking Metrics Per Variant

import time
from dataclasses import dataclass, field
from collections import defaultdict

@dataclass
class VariantMetrics:
    requests: int = 0
    total_tokens: int = 0
    total_cost: float = 0.0
    latencies: list = field(default_factory=list)
    successes: int = 0
    failures: int = 0
    tool_calls: int = 0

class ABTestTracker:
    def __init__(self):
        self.metrics = defaultdict(VariantMetrics)
    
    def record(self, variant: str, response: dict, latency: float, cost: float):
        m = self.metrics[variant]
        m.requests += 1
        m.total_tokens += response.get("usage", {}).get("total_tokens", 0)
        m.total_cost += cost
        m.latencies.append(latency)
        if response.get("choices", [{}])[0].get("finish_reason") == "stop":
            m.successes += 1
        else:
            m.failures += 1
    
    def summary(self) -> dict:
        result = {}
        for variant, m in self.metrics.items():
            p50 = sorted(m.latencies)[len(m.latencies) // 2] if m.latencies else 0
            p99 = sorted(m.latencies)[int(len(m.latencies) * 0.99)] if m.latencies else 0
            result[variant] = {
                "requests": m.requests,
                "avg_cost": m.total_cost / max(m.requests, 1),
                "total_cost": m.total_cost,
                "avg_tokens": m.total_tokens / max(m.requests, 1),
                "p50_latency_ms": p50 * 1000,
                "p99_latency_ms": p99 * 1000,
                "success_rate": m.successes / max(m.requests, 1),
            }
        return result

Statistical Significance

Don’t call a winner too early. Use proper statistical tests:

from scipy import stats
import numpy as np

def ab_test_significance(
    variant_a_scores: list[float],
    variant_b_scores: list[float],
    alpha: float = 0.05
) -> dict:
    """Two-sample t-test for A/B comparison."""
    t_stat, p_value = stats.ttest_ind(variant_a_scores, variant_b_scores)
    
    mean_a = np.mean(variant_a_scores)
    mean_b = np.mean(variant_b_scores)
    lift = (mean_b - mean_a) / mean_a * 100
    
    # Minimum sample size calculation
    effect_size = abs(mean_a - mean_b) / np.std(variant_a_scores + variant_b_scores)
    min_sample = max(30, int((2.8 / effect_size) ** 2)) if effect_size > 0 else 1000
    
    return {
        "variant_a_mean": mean_a,
        "variant_b_mean": mean_b,
        "lift_pct": lift,
        "p_value": p_value,
        "significant": p_value < alpha,
        "min_sample_size": min_sample,
        "current_sample": len(variant_a_scores) + len(variant_b_scores),
    }

What to A/B Test

Prompt Variations

prompts = {
    "concise": "Answer concisely. Maximum 2 sentences.",
    "detailed": "Provide a thorough, detailed explanation with examples.",
    "structured": "Answer using markdown with headers and bullet points.",
    "chain_of_thought": "Think step by step before answering."
}

Context Window Strategies

context_strategies = {
    "full_history": "Include all previous messages",
    "sliding_window": "Last 10 messages only",
    "summarized": "Summarize history every 5 turns",
    "rag_augmented": "Retrieve relevant past conversations"
}

Model Variants

model_variants = {
    "fast_cheap": "openai/gpt-4o-mini",
    "balanced": "openai/gpt-4o",
    "premium": "anthropic/claude-sonnet-4-20250514",
    "local": "ollama/llama3.1"
}

Production Dashboard

# Grafana/Prometheus metrics
from prometheus_client import Counter, Histogram, Gauge

VARIANT_REQUESTS = Counter(
    "agent_ab_requests_total",
    "Total requests per variant",
    ["variant"]
)

VARIANT_LATENCY = Histogram(
    "agent_ab_latency_seconds",
    "Response latency per variant",
    ["variant"],
    buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

VARIANT_COST = Gauge(
    "agent_ab_cost_dollars",
    "Cumulative cost per variant",
    ["variant"]
)

VARIANT_SATISFACTION = Gauge(
    "agent_ab_user_satisfaction",
    "Average user satisfaction score",
    ["variant"]
)

A/B testing agents requires patience—agent quality metrics are noisy and effects are small. Run tests for at least 1000 conversations per variant before drawing conclusions, and always monitor cost alongside quality.