Retry Strategies That Respect Model and Rate Limits

A naive retry loop is the worst thing you can do to a rate-limited model API. The provider throttles you because demand exceeds quota; a tight retry loop converts one rejected request into four, pushes you deeper into the limit, and turns a one-second blip into a cascade. Retry strategy for LLM workloads has three jobs: retry only what is worth retrying, back off in a way the provider’s budget recognizes, and fail over to a different deployment instead of hammering the same one. LiteLLM implements all three in its Router — worth understanding even if you build your own, because the failure modes it encodes are universal.

Retry Only the Retriable

Start with the classification, because it drives every knob after it. LiteLLM’s Router treats errors in three buckets:

• Rate limits (429) — retry, but with exponential backoff. The docs are explicit: “For RateLimitError we implement exponential backoffs. For generic errors, we retry immediately.”
• Transient infra errors — connection drops, 503s, timeouts: retriable, fast retry acceptable.
• Deterministic errors — 401 auth failures, 404 unknown deployments, 408s, content-policy violations: retrying is useless and, in the 401 case, actively harmful (it burns quota and can trip account-level lockouts).

The per-error-type control is RetryPolicy, one field per exception class plus a default:

import litellm
from litellm.router import RetryPolicy, AllowedFailsPolicy

retry_policy = RetryPolicy(
    RateLimitErrorRetries=3,               # backoff and retry rate limits
    ServiceUnavailableErrorRetries=2,      # transient infra
    TimeoutErrorRetries=2,
    InternalServerErrorRetries=2,
    AuthenticationErrorRetries=0,          # never — fix the key, not the loop
    NotFoundErrorRetries=0,                # a deleted response id stays deleted
    BadRequestErrorRetries=1,              # one shot in case it's transient validation
    ContentPolicyViolationErrorRetries=0,  # the model said no; retrying won't change it
    DefaultRetries=1,                      # anything not listed above
)

router = litellm.Router(
    model_list=[...],
    retry_policy=retry_policy,
    num_retries=3,
    retry_after=5,   # minimum floor: wait at least 5s before retrying
)

The precedence rule matters: the most specific field wins. NotFoundErrorRetries governs any 404 regardless of how the provider’s error body mapped, BadRequestErrorRetries covers the rest of the invalid-request 4xxs, and DefaultRetries applies last — so a policy that only sets DefaultRetries will retry your 404s too. Set the zero fields deliberately.

One Retry Loop, Not Two

The subtlest production bug in LLM retries is double-looping. LiteLLM documents the trap directly: “num_retries is not the same knob as max_retries. num_retries is LiteLLM’s own retry loop, while max_retries is the provider SDK’s internal retry count.” When a call goes through the router, LiteLLM owns retries and pins the provider client to max_retries: 0 — deliberately, “it is what stops a deployment num_retries: N from being applied twice and turning one request into (1 + N) ** 2 upstream calls.”

If you wrap a provider SDK that has its own retry client, apply the same discipline: exactly one layer retries, everything inside it is pinned to zero retries. And make the retry count request-scoped when you can. LiteLLM’s own precedence ranking is a good model — x-litellm-num-retries header, then request-body num_retries, then deployment config, then the global default — so a caller can lower retries to 0 for a latency-sensitive path regardless of the fleet default.

Cool Down the Deployment, Not Just the Request

Retrying the same deployment on a 429 is often pointless — that deployment’s quota is spent. LiteLLM’s answer is cooldowns: after too many failures, the individual deployment is temporarily removed from the pool while healthy peers keep serving. The knobs:

router = litellm.Router(
    model_list=model_list,
    allowed_fails=1,     # cool down after >1 failure in a minute
    cooldown_time=100,   # stay out of rotation for 100 seconds
)

Default cooldown triggers are sensible and worth copying even in a homegrown system: a 429 triggers cooldown immediately, a failure rate above 50% in the current minute triggers it, and non-retryable errors (401, 404, 408) trigger it — all with a short default cooldown that lets the deployment rejoin rotation quickly. The unit of isolation is the deployment (a single entry in the model list with its own litellm_params and hashed model_id), not the whole model group — one rate-limited regional endpoint should not take out its healthy siblings. For per-endpoint strictness, scope an AllowedFailsPolicy to one deployment via model_info, e.g. cooling down a flaky third-party proxy after its first RateLimitError while the rest of the fleet keeps the router-wide tolerance.

Fail Over Along a Defined Topology

Once cooldowns exist, retries become a routing problem: send the next attempt somewhere else. LiteLLM gives you three escalating mechanisms, and they compose in order.

Priority order. Set order in each deployment’s litellm_params — lower is higher priority. “When a request to an order=1 deployment fails (connection error, 404, 429, etc.), the router automatically tries order=2 deployments, then order=3, and so on. Each order level gets its own set of retries before escalating.”

Weighted failover within a group. With enable_weighted_failover=True (simple-shuffle strategy, async entrypoints only), a retryable failure excludes the failed deployment’s id and re-picks from the remaining peers using their weights — for regional copies of the same model, this fails a bad region over to a healthy one before touching a different provider. Exclusions accumulate across hops, it is capped by max_fallbacks, and it deliberately does not trigger for ContextWindowExceededError or ContentPolicyViolationError, which keep their own dedicated fallback paths.

Cross-group fallbacks. Only after the group is exhausted does the router consult fallbacks — typically a different model family, where “retry” becomes “answer with a smaller model instead of failing the user.”

One more rate-limit-respecting control worth stealing: max_parallel_requests. Rather than queueing over-quota requests, LiteLLM rejects overflow immediately with a 429 raised before any provider call — “so it does not count towards the deployment’s cooldown” — and derives sensible per-deployment caps from declared rpm/tpm when you don’t set one. Fast rejection at the edge is kinder to callers than silent queueing that shows up as 200-second latencies.

Prevent Retries You’ll Regret

The best retry is one you never need. Two LiteLLM patterns map directly onto agent workloads:

Pre-call context checks. With enable_pre_call_checks=True, the router filters out deployments whose context window cannot fit the incoming messages before sending anything. Truncation is a design decision; a ContextWindowExceededError discovered after the request wastes both the attempt and the retry budget.

Idempotency awareness. Retrying a chat completion is safe; retrying a tool with side effects may not be. In agent loops, restrict aggressive retries to the model calls and keep tool execution retry policy explicit and usually stricter — a 429’d model call should back off, while a payment.charge tool that timed out needs reconciliation, not a second attempt.

Conclusion

Respectful retries come from four decisions made in advance: which errors are retriable (encode it in a RetryPolicy with explicit zeros), where the single retry loop lives (pin everything inside it to zero retries), how long to stay away (exponential backoff on 429s, deployment cooldowns with allowed_fails and cooldown_time), and where the next attempt goes (order tiers, weighted failover, then cross-group fallbacks). LiteLLM’s Router ships all of it — the discipline transfers whether you use the library or not. Build the policy once, test it by injecting 429s and timeouts, and let your retry behavior be the thing that keeps you inside the rate limit instead of the thing that keeps knocking on its door.