LLM-as-a-Judge for Agent Outputs: Strengths and Pitfalls

One LLM evaluates another LLM. It sounds circular — but it works. LLM-as-a-Judge has become the dominant evaluation paradigm for language models, powering MT-Bench, AlpacaEval, Chatbot Arena, and RewardBench. The question isn’t whether to use it, but how to use it without fooling yourself.

Three Evaluation Patterns

Pattern	Description	Best For
Pairwise comparison	Judge sees two outputs, picks winner	A/B testing, prompt iteration
Single-output with reference	Score against known correct answer	QA, summarization
Single-output without reference	Apply rubric criteria, no comparison	Production monitoring

Key insight: LLMs are better at discriminating between options than generating absolute scores. Pairwise comparisons produce more reliable assessments than pointwise scoring.

The Strengths

Scalability — 15-100x cheaper than human annotation. Gemini 2.5 Flash achieves 71% agreement with humans at ~$0.001/evaluation vs. ~$0.015 for frontier models.

Semantic understanding — Unlike BLEU/ROUGE, LLM judges understand meaning, context, and nuance. They evaluate subjective qualities: helpfulness, tone, safety, coherence.

Flexibility — Judges evaluate custom criteria via natural language rubrics. Reference-based or referenceless depending on ground truth availability.

Human alignment — GPT-4 achieves >80% agreement with human preferences on MT-Bench.

The Pitfalls: Systematic Biases

Verbosity Bias (The Dominant One)

Pro/Llama/Flash prefer longer responses (+0.24 to +0.44 bias score). Claude prefers concise responses (-0.12). GPT-4o is essentially neutral (-0.04).

Critical finding: On truncation pairs where longer = genuinely more complete, ALL models correctly prefer the longer response (88-100% accuracy). Judges can distinguish filler from substance — but it requires calibration.

Style Bias (The Most Severe)

0.40-0.76 baseline across models, overwhelmingly favoring markdown-formatted responses. Human annotators preferred markdown 57% of the time, while 4/5 judges preferred it 73-97%. A gap of 17-40 percentage points.

Self-Preference Bias

LLMs exhibit bias toward outputs from their own model family (51.4%-86.2% self-preference rates). When test data AND evaluators are from the same model, rankings are systematically inflated.

The Consistency-Validity Paradox

High test-retest reliability (>0.95) can coexist with severe position bias (>0.10). A judge that deterministically favors position A achieves perfect test-retest but maximum position bias. Reliability ≠ Validity.

Debiasing Strategies (Ranked)

Strategy	Description	Effect
Position Swap	Run judge twice with A/B order reversed; tie on disagreement	+4.7pp
Chain-of-Thought	Step-by-step reasoning before verdict	+7.3pp
Calibrated Rubric	Structured 5-criteria rubric	Moderate
Combined Budget	Position swap of merged CoT+rubric prompt	+11.5pp

Code: Bias-Mitigated Pairwise Judge

import anthropic

def pairwise_judge(question: str, response_a: str, response_b: str) -> dict:
    rubric = """Evaluate based on:
    1. Accuracy (1-5): Factual correctness
    2. Relevance (1-5): How well it answers the question
    3. Completeness (1-5): Coverage of key points
    4. Clarity (1-5): Readability and organization
    5. Reasoning depth (1-5): Quality of explanation"""
    
    prompt = f"""You are an impartial judge. Analyze step by step.
{rubric}

Question: {question}
Response A: {response_a}
Response B: {response_b}

Provide analysis and verdict as JSON:
{{"analysis": "...", "verdict": "A" | "B" | "tie"}}"""
    
    client = anthropic.Anthropic()
    
    # Run twice with swapped positions
    result1 = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    
    prompt_swapped = prompt.replace("Response A:", "TEMP").replace(
        "Response B:", "Response A:"
    ).replace("TEMP", "Response B:")
    
    result2 = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt_swapped}]
    )
    
    import json
    v1 = json.loads(result1.content[0].text)
    v2 = json.loads(result2.content[0].text)
    
    # Tie on disagreement
    if v1["verdict"] != v2["verdict"]:
        return {"verdict": "tie", "reason": "Disagreement after position swap"}
    
    return v1

Code: Custom Evaluation with DeepEval

from deepeval import evaluate
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

test_case = LLMTestCase(
    input="Summarize our refund policy.",
    actual_output="Customers can return shoes within 30 days.",
    expected_output="Customers can return eligible shoes within 30 days.",
)

correctness = GEval(
    name="Correctness",
    evaluation_steps=[
        "Check whether actual output contradicts expected output.",
        "Penalize missing eligibility conditions.",
        "Do not penalize harmless wording differences.",
    ],
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.EXPECTED_OUTPUT,
    ],
    threshold=0.7,
)

evaluate(test_cases=[test_case], metrics=[correctness])

Validation Protocol

1. Smoke test — 1 model, 1 benchmark → agreement check
2. Bias audit — Test on controlled datasets for position, verbosity, style, self-preference
3. Consistency — Test-retest reliability
4. Cost-tier coverage — Different model tiers for optimization
5. Full validation — Cross-benchmark, cross-model evaluation quarterly

The Takeaway

LLM-as-a-Judge is powerful but requires deliberate debiasing. Use position swap + chain-of-thought + calibrated rubric for the strongest results. Always validate against 200-500 human-labeled cases quarterly, targeting correlation >0.85. Use different judge families from the evaluated model to avoid self-preference.

💡 Gemini Flash with debiasing costs ~$0.001/evaluation — 15x cheaper than frontier models with comparable accuracy.