Failure Injection for LLM Calls: Simulating Model Outages Before They Simulate Themselves

Your agent’s retry logic, fallback model, and degraded-mode path have never run in production. They also never ran in CI, because your tests mock the LLM with a function that always returns a perfect JSON object. The first provider outage then exercises thousands of lines of untested recovery code — live, on customer traffic, at the worst possible moment.

The fix is to stop testing the happy path alone and start injecting failures into LLM calls: timeouts, rate limits, server errors, streams cut mid-token, malformed JSON, content-filter blocks. This is a tutorial for doing it deliberately — a failure taxonomy, three injection points (mock transport, monkeypatched client, chaos gateway), deterministic seeds for reproducible failures, what to actually assert, how to run it as a CI suite, and a small sample of injected chaos in production.

Why standard tests never catch resilience bugs

Unit tests mock at the wrong layer. A mock that raises Exception("boom") when you feel like it tests that your code catches that exception; it does not test that your HTTP timeout fires before the provider’s, that a 429 with a Retry-After header is honored rather than hammered, or that a stream which dies at token 40 of a structured-output response does not get recorded as a complete answer. The failure modes that hurt are transport- and format-level, and happy-path mocks delete exactly that layer from the test.

There is a second, subtler problem: resilience bugs are intermittent by construction. A retry that works on attempt one but triple-charges on attempt three, a fallback that engages only when the primary fails twice — these paths never execute in a suite where every call succeeds.

A failure taxonomy for LLM calls

Be systematic. Every incident in this table has taken down a real agent system:

Failure	How it looks on the wire	Why it breaks naive code
Connect timeout	TCP never establishes	No exception type handled; request hangs until some timeout
429 rate limit	HTTP 429 + Retry-After	Retrying immediately makes it worse; ignoring it fails the run
5xx	HTTP 500/502/503	Ambiguous: the model call may or may not have been executed
Slow-loris latency	200 OK, tokens after 30s	No timeout configured; worker pool deadlocks behind slow calls
Stream cut mid-token	SSE ends without [DONE]	Partial JSON accepted as complete; truncated tool calls
Malformed / truncated JSON	Model returns 90% of a JSON object	Structured-output parsing crashes instead of retrying with a repair prompt
Content filter block	Provider refuses the request	Treated as retryable; loops on a permanently blocked input
Region / deployment outage	Repeated 5xx from one endpoint	No fallback path, or fallback that shares the failure domain

Each row deserves at least one test, and the assertions differ per row: some should trigger a retry, some a fallback, and some — the content filter, the bad request — should fail fast without retrying at all.

Three injection points

1. Mock transport (preferred for CI)

If your client speaks HTTP (the OpenAI SDK, Anthropic SDK, LiteLLM — all HTTP under the hood), inject failures at the transport, not in your business logic. httpx.MockTransport swaps the transport handler, so every layer above it — the SDK’s retry, your timeout wrapper, your fallback chain — runs unmodified:

# chaos.py — a deterministic, scripted failure injector for httpx-based clients.
import httpx

class FailurePlan:
    """Ordered script of behaviours. Each call consumes the next entry."""
    def __init__(self, steps):
        self.steps = list(steps)
        self.calls = 0

    def __call__(self, request: httpx.Request) -> httpx.Response:
        step = self.steps[min(self.calls, len(self.steps) - 1)]
        self.calls += 1
        return step(request)

def timeout_after(seconds: float):
    def step(request):
        raise httpx.ConnectTimeout(f"injected: slept {seconds}s then gave up")
    return step

def status(code: int, body: dict | None = None, retry_after: int | None = None):
    def step(request):
        headers = {}
        if retry_after is not None:
            headers["Retry-After"] = str(retry_after)
        return httpx.Response(code, json=body or {"error": {"message": "injected"}}, headers=headers)
    return step

def ok(body: dict):
    def step(request):
        return httpx.Response(200, json=body)
    return step

def truncated_json(good_body: dict):
    """Return a 200 whose content is cut mid-string — the classic stream death."""
    def step(request):
        import json as _json
        content = _json.dumps(good_body)
        cut = content[: len(content) // 2]
        return httpx.Response(200, content=cut)
    return step

# Wire it into any client that accepts a transport:
plan = FailurePlan([
    status(429, retry_after=1),            # call 1: rate limited
    timeout_after(30),                     # call 2: hangs, times out
    ok({"choices": [{"message": {"content": "done"}}]}),  # call 3: succeeds
])

That is the shape that exercises a bounded retry loop: fail twice, then succeed. Because the plan is an explicit list, the test is deterministic: no random.random() < 0.3 branches that make a red run unexplainable.

2. Library-level monkeypatching

When the client does not let you swap the transport, patch the innermost call and raise the real exception types from your provider’s error module (litellm.RateLimitError, openai.APITimeoutError, anthropic.InternalServerError). Using the real exception classes matters: your fallback logic almost certainly branches on exception types, and a generic RuntimeError will not hit those branches. This is the honest middle ground — less faithful than a transport mock (the SDK’s own retry may be bypassed) but far more faithful than stubbing your service function.

3. Chaos gateway: LiteLLM Router with a booby-trapped deployment

For integration tests and production sampling, put a real gateway in front and give it a deliberately bad deployment. LiteLLM’s Router is built for exactly this: deployments fail over by order, a 429 triggers an immediate cooldown by default (5 seconds out of rotation), allowed_fails / AllowedFailsPolicy controls how many failures per minute before a deployment is cooled, and RetryPolicy sets per-error-type retry counts (TimeoutErrorRetries, RateLimitErrorRetries, NotFoundErrorRetries, …) so 404s and auth errors fail fast while timeouts get retried. Add a deployment with a blackhole base_url (connect timeouts), one with a bad API key (auth errors), and one healthy deployment — then assert the router lands on the healthy one:

from litellm import Router

model_list = [
    {"model_name": "assistant", "litellm_params": {
        "model": "openai/gpt-6.1-sol", "api_key": "sk-broken",
        "api_base": "http://127.0.0.1:9",          # nothing listens: fast 5xx/connect error
        "order": 1}},
    {"model_name": "assistant", "litellm_params": {
        "model": "openai/gpt-6-luna", "api_key": "sk-good",
        "api_base": "http://127.0.0.1:9",          # blackhole for timeout tests
        "order": 2}},
]
router = Router(model_list=model_list, num_retries=1,
                cooldown_time=30, allowed_fails=1)

The same router configuration doubles as a production chaos tool: flag a small percentage of real requests to prefer the degraded deployment (or set per-request num_retries/routing metadata), and watch your fallback path execute against real traffic at a controlled rate.

Deterministic seeds and repro

Chaos tests that cannot be reproduced are just flakiness with better marketing. Three rules:

• Script, don’t sample, in CI. The FailurePlan above is a list, not a random draw. Reproduce a failure by re-running the same plan.
• Seed the model too. If your test asserts on model output after a retry, pin temperature to 0 or use recorded responses; otherwise the failure is deterministic but the recovery output is not.
• Log the plan position on failure. When an assertion fires, print which step of the plan produced the request that failed (plan.calls), so the red run tells you the story instead of just the outcome.

What to assert

The recovery behaviour is the test. After running the agent against a failure plan, assert:

• The fallback model was chosen. Inspect the response metadata or a span attribute recording which deployment served the request — not just that some text came back.
• Retries were bounded. The mock transport’s plan.calls equals exactly the expected number (e.g. 2 failures + 1 success), not “at most 5”. Unbounded retries are how a 429 becomes a $4,000 bill.
• The response degraded but stayed correct. A fallback to a smaller model should still produce schema-valid output; a truncated stream should be detected, not accepted. Assert on the Pydantic output type validating, not on string equality.
• No duplicate tool calls after a retry. This is the expensive one: if the model call was retried after the model emitted a tool call that executed, the retry can emit it again. Count executions of your tool (a simple counter in the fake) and assert it ran once, or verify the idempotency key passed downstream was stable across the retry.
• Fast-fail paths fast-failed. Content filters and 4xx errors should produce zero or one upstream calls, not five.

Wiring it into CI as an eval suite

Run the failure matrix on every PR, alongside your quality evals:

import pytest

CASES = {
    "rate_limited_then_ok":   FailurePlan([status(429, retry_after=1), status(200)]),
    "timeout_then_fallback":  FailurePlan([timeout_after(30), timeout_after(30), status(200)]),
    "truncated_stream":       FailurePlan([truncated_json(canned), status(200)]),
    "content_filter_no_retry":FailurePlan([status(400, {"error": {"code": "content_filter"}})]),
}

@pytest.mark.parametrize("name,plan", CASES.items())
async def test_agent_survives(name, plan, monkeypatch):
    client = build_client_with_transport(plan)
    result = await run_agent(client, "summarize ticket #42")
    assert result.output is not None
    assert plan.calls <= MAX_UPSTREAM_CALLS[name]

The suite is cheap — no real API calls, no network, milliseconds per case — and it fails the build when somebody’s refactor deletes a retry branch. Keep a separate, scheduled (nightly, not per-PR) job for live chaos against a staging deployment with real credentials; that one catches what mocks cannot, like provider-specific error bodies.

Sampling chaos in production

The last mile: inject a tiny, gated percentage of failures into live traffic — 0.1% of calls get a synthetic 500 or a delayed first token — and measure whether the system recovers the way CI says it will. Guardrails that keep this from being an outage-with-extra-steps:

• A kill switch (config flag read per request) and an env-scoped allowlist of traffic — never inject into payment, refund, or safety-critical tool paths.
• Inject before the fallback path, so recovery has somewhere to go; injecting a failure into your only model with no fallback just creates errors.
• Alert on the injected-failure recovery rate, not just the raw error rate. The metric that matters: “percentage of injected failures that ended in a successful degraded response.”

Instrument every injected failure as a normal error span with an attribute like chaos.injected=true (OpenTelemetry makes this a one-liner on the span), so you can slice recovery metrics without a separate dashboard.

Checklist

• Build a failure taxonomy table for your agent and give every row a test.
• Inject at the transport layer (httpx.MockTransport) so SDK retries and timeouts run unmodified; monkeypatch with real provider exception types when you cannot.
• Use real exception classes and real error shapes; generic Exception tests nothing.
• Script failures as ordered plans, seed model outputs, and log plan position on failure.
• Assert on fallback selection, bounded retry counts, schema-valid degraded output, and exactly-once tool execution — not just “it returned text”.
• Run the matrix in CI per PR; run live chaos against staging nightly; sample synthetic failures in production behind a kill switch, never on money paths.

Further reading

• LiteLLM — gateway with router cooldowns, per-error-type RetryPolicy, and deployment failover
• LiteLLM — Router: load balancing and reliability — num_retries, allowed_fails, cooldowns, weighted failover
• OpenTelemetry documentation — tagging injected failures on spans for production chaos measurement