A/B Testing Prompts in Live Applications

Prompt engineering is product development, but too often it’s treated like a vibe: rewrite the system prompt, watch a few outputs, ship it. “It feels better” is how a prompt change quietly drops answer quality for a month before anyone notices. A/B testing prompts is how you turn prompt changes into decisions with evidence.

In this post, you will learn how to run a proper prompt A/B test in a live application: deterministic bucketing, model parity, controlled rollout, and metrics that separate real signal from stochastic noise.

The core problem: LLMs are noisy

Run the same prompt twice and you get different text. That noise is the #1 reason prompt A/B tests lie — unless you control it. Two rules of thumb:

1. One change at a time. Differs in wording and added context and temperature → you can’t attribute the outcome.
2. Control the temperature. For tests, pin the sampling to a fixed temperature (often 0 or a fixed low value) on both variants so output variance doesn’t swamp your effect.

Deterministic bucketing

Assign users to variants deterministically so (a) a user always sees the same variant and (b) your metric code can group by variant. Hash the user ID, don’t use a random coin flip:

import hashlib

def variant_for(user_id: str, variants: list[str]) -> str:
    digest = hashlib.sha256(user_id.encode()).digest()[0]
    return variants[digest % len(variants)]

Store the assignment in the request log, not just the response — you’ll want it at analysis time.

The two-layer test: offline first, online second

Never jump straight to live traffic. Layer 1 is offline: run both prompts over a fixed eval set and measure with a judge LLM or reference answers. If A isn’t clearly better offline, don’t spend the traffic.

import pandas as pd
from google import genai

client = genai.Client()
evalset = pd.read_csv("eval_queries.csv")   # query, reference_answer

def judge(prompt_variant, query, answer):
    return client.models.generate_content(
        model="gemini-2.5-pro",
        contents=f"Rate 0-1 how well this answer satisfies the query.\n"
                 f"QUERY: {query}\nANSWER: {answer}",
        config={"temperature": 0},
    ).text

Run variant A and B over all queries at temperature=0, collect judge scores, and compare distributions (mean, pass-rate at 0.8). Layer 2 is the live test described below — and you run it because offline evals miss real-world context like conversation history and RAG injection.

The live test in production

When you go live, keep the variants behind a feature flag and route by the deterministic bucket:

PROMPT_VARIANTS = {
    "control": SYSTEM_PROMPT_V1,
    "candidate": SYSTEM_PROMPT_V2,
}

@app.post("/api/chat")
def chat(user_id: str, user_message: str):
    variant = variant_for(user_id, ["control", "candidate"])
    messages = [{"role": "system", "content": PROMPT_VARIANTS[variant]},
                *history(user_id),
                {"role": "user", "content": user_message}]
    response = client.models.generate_content(
        model="gemini-2.5-pro", contents=messages, config={"temperature": 0}
    )
    log_row(user_id=user_id, variant=variant, message=user_message,
            response=response.text, latency_ms=..., usage=response.usage_metadata)
    return {"answer": response.text, "variant": variant}

Log everything: variant, latency, tokens, and the exact inputs. You can’t analyze a test you didn’t instrument.

Metrics: quality over clicks

“Which variant does the user like more?” is rarely measurable directly. Use proxies that matter:

• Task success: did the user complete the action (accept the suggestion, reach checkout)? The strongest signal when it exists.
• Follow-up rate: a lower rate of clarifying follow-ups usually means the answer was clearer.
• Latency and tokens: a “better” prompt that costs 2x tokens isn’t better — track both per variant.
• Judge-LLM score on sampled responses: reuse your offline judge on a random sample of live responses for quality.

Know your sample size

LLM metrics move slowly; a week of traffic might not be enough. A quick power calculation keeps you honest:

from statsmodels.stats.power import NormalIndPower

# Detect a 2% lift in pass-rate with 95% confidence / 80% power
n = NormalIndPower().solve_power(
    effect_size=0.2, alpha=0.05, power=0.8, ratio=1.0, alternative="two-sided"
)
print(f"needed per variant: {n:.0f}")

Stop the test early only for safety issues (a hard regression, or a crash) — not because the numbers “feel” done.

Putting It All Together

The full harness — deterministic bucketing, flag-gated variants, structured logging, and the offline judge — is in this gist. Wire it up, ship the candidate behind the flag at 10% of traffic, and let the metric decide.

Conclusion & Next Steps

You can now A/B test prompts with controlled noise, real-world context, and objective metrics. Next: automate the offline judge into CI so every prompt PR runs the eval before review, add a dashboard that tracks variant metrics over time, and consider multivariate tests — but only after the single-variable pipeline is stable.

References / Sources

• Gemini API prompts and parameters. https://ai.google.dev/gemini-api/docs
• FastAPI request handling. https://fastapi.tiangolo.com
• Statistical power analysis (statsmodels). https://www.statsmodels.org