Rate-Limiting Agent Tools: 429s, Throttling, and Graceful Degradation

An agent that retries a 429 Too Many Requests as fast as it can is an agent that turns a momentary limit into an outage. Rate limits are unique among errors: the correct response is to stop, and the correct place to handle them is usually one layer below the agent’s reasoning loop.

There are three different limits wearing the same status code, and conflating them is the first mistake:

1. Provider limits — RPM/TPM quotas from OpenAI, Anthropic, Google, your Azure deployment.
2. Your gateway’s limits — quotas you set per key, team, model, or tenant, which you enforce as a hard cap.
3. Per-tool limits — the search API, CRM, or payment provider your tools call, each with its own throttle.

The agent loop needs to behave correctly for all three.

What a 429 actually tells you

A well-formed 429 carries a retry hint — Retry-After in seconds, or provider-specific headers like x-ratelimit-reset-requests. Honoring it is non-negotiable: sleeping for exactly the window the server asked for beats any backoff you invent.

LiteLLM’s gateway responses are explicit about the cause, which makes triage mechanical:

Error code: 429 - {'error': {'message': 'Model capacity reached for my-fake-model.
Priority: None, Rate limit type: tokens, Model TPM: 60, Model RPM: not configured,
Remaining: 0', 'type': 'throttling_error', 'param': None, 'code': '429'}}

type: throttling_error with Rate limit type: tokens means you exhausted a token budget; requests means a request-frequency cap. The two want different responses — a token-limited run should reduce output length, a request-limited run should reduce call count.

Client-side: backoff that respects the window

import asyncio, random
from openai import APIStatusError

async def call_with_limits(fn, *, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return await fn()
        except APIStatusError as e:
            if e.status_code != 429 or attempt == max_attempts - 1:
                raise
            retry_after = e.response.headers.get("retry-after")
            delay = float(retry_after) if retry_after else min(2 ** attempt + random.random(), 30)
            await asyncio.sleep(delay)   # never a blocking sleep in an agent loop

Rules that matter:

• Exponential backoff with jitter, capped. Without jitter, every concurrent agent retries at the same instant — the thundering herd that made the 429 worse.
• A retry budget. Cap total attempts and total time spent waiting. An agent with a 60-second task deadline should not burn 55 of it in backoff.
• Async sleep. A blocking sleep stalls the whole agent runtime, including unrelated tasks.
• Classify before retrying. 429 is retryable; 401/403/400 are not. Retrying a rejected request with a bad schema just burns quota.

Gateway-side: make the limit a policy, not a surprise

This is where LiteLLM (the cited gateway for this topic) earns its place. Instead of discovering limits through provider errors, you declare them and the gateway blocks before the request ever reaches the provider.

Limits attach at several levels — key, team, project, organization, and model:

curl http://0.0.0.0:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"max_parallel_requests": 10, "tpm_limit": 200000, "rpm_limit": 40}'

Per-model caps on a single key:

-d '{"model_rpm_limit": {"gpt-5.6-terra": 2}, "model_tpm_limit": {"gpt-5.6-terra": 1000}}'

Responses advertise the remaining budget, which lets a well-behaved client slow down before it hits the wall:

x-litellm-key-remaining-requests-gpt-5.6-terra: 1
x-litellm-key-remaining-tokens-gpt-5.6-terra: 179

For deployment-level enforcement (each deployment declares rpm/tpm, the router refuses excess), enable the pre-call check:

model_list:
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
      rpm: 60      # 60 requests per minute
      tpm: 90000   # 90k tokens per minute

router_settings:
  optional_pre_call_checks:
    - enforce_model_rate_limits

With that on, overload returns a local 429 instead of a provider error — cheaper, faster, and correct. In multi-instance deployments, back the counters with Redis (redis_host/redis_port under router_settings); otherwise each proxy instance enforces its own private copy of the limit and the aggregate exceeds the real quota.

Two knobs people miss:

• num_retries: 0 surfaces the 429 to the caller instead of having the router retry it — the right choice when you own the retry policy and don’t want hidden attempts eating quota.
• max_parallel_requests vs derived limits. Setting rpm implicitly derives a parallel-request cap from it. If you find legitimate traffic rejected with “Deployment has all max_parallel_requests slots in use”, raise max_parallel_requests explicitly or the rpm/tpm it was derived from.

Graceful degradation: what the agent does when the limit is real

Backoff handles transient limits. When the budget is genuinely exhausted, the run needs a plan that is not “wait longer”:

1. Shed load deliberately. Rank queued work by priority. Agent tasks spawned by a user waiting on screen outrank a nightly enrichment sweep; pause the sweep instead of degrading both.

2. Fall back across models. Route to a cheaper or roomier model for the same task. Most agent workloads tolerate a quality dip on routine steps far better than a hard failure — and a fallback chain is configured once, not coded per call.

3. Serve from cache. Semantic or tool-result caching turns repeat requests into zero-quota work. If the last turn’s tool output or a near-identical prompt was seen recently, answer from cache.

4. Degrade the feature, not the product. “Summarize this thread” can return the last cached summary with a note; a chat reply can be shorter (fewer output tokens = less TPM); a non-critical enrichment step can be skipped and reconciled later.

5. Queue with a deadline. A bounded queue absorbs a burst; a deadline converts it into a clear, honest message. Unbounded queues just move the 429 to a later, angrier moment.

6. Tell the user. Agents should propagate a typed “rate limited until T” state up to the UI rather than emitting a generic failure. The UX for “try again in 40 seconds” is trivial; the UX for “something went wrong” is not.

Per-tool throttling inside the agent loop

Your own tools need the same treatment, and they are often stricter than the model API. Practical shape:

• A token bucket per tool (not per call site), so N parallel sub-agents share one budget.
• Concurrency caps with a semaphore — many APIs limit parallel in-flight requests rather than frequency.
• Bulkheads: one full tool queue must not block unrelated tools.
• 429 as a first-class tool error the model can see, so it can choose to do something else instead of retrying blindly. Wrapping the error as "Rate limited by search API, retry in 30s" lets a smart agent plan around it; a stack trace does not.

Observability

You cannot tune what you cannot see. Track per-tool and per-model: 429 count by source, time spent in backoff, queue depth, requests rejected by your gateway vs the provider, and cache hit rate. A rising ratio of gateway-rejections means your declared limits are too low; a rising ratio of provider-rejections means they’re too high — or something is bypassing the gateway.

Alert on sustained 429 rates and on backoff-time as a share of total latency. A single spike is a burst; 5% of wall-clock time spent sleeping is a capacity problem.

Common mistakes

• Retrying storms — no jitter, no cap, all workers retrying in lockstep.
• Retrying the wrong errors — auth and validation failures are not transient.
• Hidden gateway retries eating the retry budget you thought you had.
• One global limiter for every tenant — a noisy neighbor starves everyone.
• Treating 429 as a bug in dashboards rather than a signal with a target.
• Forgetting tools — all attention on model quotas while the CRM API quietly 429s.

Checklist

• Honor Retry-After first; fall back to capped exponential backoff with jitter.
• Enforce limits at the gateway before the provider sees the traffic; use Redis when running multiple instances.
• Read the x-litellm-key-remaining-* headers and pre-emptively slow down.
• Define degradation explicitly: priority shedding, model fallback, cache, queue-with-deadline.
• Rate-limit tools per function, with concurrency caps and typed errors.
• Track 429s by source and alert on sustained throttling.

Sources: LiteLLM, LiteLLM docs — budgets and rate limits, LiteLLM docs — load balancing and model rate limits.