Backpressure for Agent Ingress: Shaping Load at the LLM Gateway
How to apply real backpressure at the ingress of an agent system with LiteLLM: bounded queues, rpm/tpm limits, load shedding, budget-aware admission, and correct 429 handling.
Published on • November 16, 2026
AI Assistant

Your agent works. Then someone points it at a queue of 10,000 documents, every document spawns a sub-agent, and every sub-agent fires five model calls. Within seconds your gateway is saturated, latency climbs from 800ms to 90 seconds, retries pile on retries, and the whole system collapses under its own enthusiasm. The provider’s 429s are not the problem — they are the symptom you failed to design for.
In this post, you will learn how to shape load at the ingress of an agent system using LiteLLM as an LLM gateway: bounded queues and why queue depth is your best signal, token-bucket versus sliding-window limiting, per-key and per-model rpm/tpm limits, priority classes and load shedding, correct client-side 429 handling, and budgets as the final admission control. You’ll ship a runnable proxy config and a Python client that behaves under load.
Prerequisites
- Python 3.10+ and
pip install 'litellm[proxy]' openai httpx - A provider API key (OpenAI, Anthropic, Azure, or any of the 100+ LiteLLM supports)
- Familiarity with HTTP status codes and basic concurrency (
asyncio.Semaphore) - Optional but recommended: Redis, if you run more than one proxy instance
Where Backpressure Belongs: The Ingress
Backpressure is the discipline of telling producers “slow down” before work accumulates unboundedly. In an agent system the natural choke point is the LLM gateway — every model call funnels through it, so it’s the one place where admission control is both possible and effective.
agent fleet ingress (backpressure here) providers
┌─────────────┐ ┌───────────────────────────────────┐ ┌──────────┐
│ orchestrator│ HTTP │ LiteLLM proxy │ │ OpenAI │
│ └ subagents├───────►│ 1. auth: virtual key lookup │───────►│ Azure │
│ └ subagents│ │ 2. admission: rpm/tpm/parallel │ rpm │ Anthropic│
│ └ subagents│ │ └ bounded queue + shed │ tpm │ Bedrock │
└─────────────┘ │ 3. router: pick deployment │ └──────────┘
▲ │ 4. budget check (reservation) │
│ 429 + retry-after│ 5. post-request: usage + spend │
└─────────────────┴───────────────────────────────────┘
Note what the diagram forbids: unbounded queues. A queue with no limit isn’t resilience, it’s deferred failure — requests age in memory until their deadlines are meaningless. Every queue in the path gets a maximum depth, and when it fills, new work is rejected or shed, not parked.
Queues, Tokens, and Windows: The Theory
Two limiting algorithms dominate, and knowing which one you’re using changes how you interpret its errors.
Token bucket. N tokens refill at a constant rate; each request consumes tokens (of request count, or of tokens). Bursts are allowed up to the bucket size. It’s simple, memory-cheap, and naturally tolerates spikey traffic — the right default when your users are interactive.
Sliding window (fixed-window counters). Count requests in the current minute, reset when the minute rolls over. LiteLLM’s limiter works this way: counters like current_rpm and current_tpm are checked and incremented per minute window, and the response carries retry-after seconds until that window resets. The trade-off is exactness at the boundary — a fixed window can admit 2× the limit across a window edge, which is why enforcement is described as exact for RPM (checked before the call) but best-effort for TPM (tokens are only known after the response lands).
That last point deserves emphasis: request counts are admission-controlled; token counts are reconciled after the fact. LiteLLM reserves tokens ahead of the call and corrects against actual usage afterwards, which is why TPM can slightly overrun under concurrency.
Finally, queue depth. When a gateway does queue — or when your client-side queue grows — depth is the earliest, most actionable signal you have. Rising depth with flat throughput means you’re saturated, not slow. Alert on depth crossing a threshold long before latency does.
Configuring LiteLLM as the Shaping Layer
LiteLLM’s proxy takes a YAML config. This one declares two deployments of the same model with hard rpm/tpm ceilings, a concurrency cap, and Redis-backed counters so multiple proxy instances agree on the limit:
model_list:
- model_name: gpt-fast
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
rpm: 60 # 60 requests per minute
tpm: 900000 # 900k tokens per minute
max_parallel_requests: 8
- model_name: gpt-fast
litellm_params:
model: azure/gpt-4o-mini-west
api_key: os.environ/AZURE_API_KEY
api_base: os.environ/AZURE_API_BASE
router_settings:
routing_strategy: simple-shuffle # weighted by rpm/tpm
num_retries: 2
cooldown_time: 30
optional_pre_call_checks:
- enforce_model_rate_limits # rpm/tpm become hard limits
redis_host: os.environ/REDIS_HOST
redis_port: 6379
litellm_settings:
drop_params: true
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
Run it with litellm --config config.yaml and point any OpenAI-compatible client at http://localhost:4000. Two behaviors matter for backpressure:
- Without
enforce_model_rate_limits,rpm/tpmonly guide routing. With it, requests over the ceiling are blocked before reaching the provider and return429with aretry-afterheader — genuine admission control rather than hopeful scheduling. - When every slot governed by
max_parallel_requestsis busy, a new request fails immediately with a 429 naming the deployment. There is no invisible wait queue; earlier LiteLLM versions silently queued and produced mysterious 200-second latencies, which is precisely the anti-pattern this design rejects.
For multi-instance deployments, Redis shares the counters; while Redis is unreachable each instance falls back to in-memory counters (fail-open), and fail_closed_rate_limit_enforcement flips that to a hard ceiling if your limits must hold during an outage.
Priority Classes and Load Shedding
Not all agent traffic deserves equal treatment. LiteLLM models this with teams: a key inherits its team’s max_parallel_requests, tpm_limit, and rpm_limit, so you can carve the ingress into priority classes.
curl -X POST 'http://localhost:4000/team/new' \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d '{"team_id": "interactive", "max_parallel_requests": 8, "rpm_limit": 120, "tpm_limit": 2000000}'
curl -X POST 'http://localhost:4000/team/new' \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H 'Content-Type: application/json' \
-d '{"team_id": "batch", "max_parallel_requests": 2, "rpm_limit": 20}'
Interactive chat gets eight slots; the nightly batch gets two. When the provider tightens its quota, you shed load from the bottom: batch jobs absorb the 429s while interactive latency stays flat. You can go further by routing budget-exhausted traffic to explicitly free models — set input_cost_per_token: 0 and output_cost_per_token: 0 on a local deployment and LiteLLM skips budget checks for it, giving you a degraded-but-alive path instead of a dead end.
Handling 429 on the Client
The gateway’s 429 is a contract: not admitted, try later, here’s when. Honor it. A client that retries on a fixed 100ms schedule is adding fuel to the fire.
import asyncio, random, httpx
async def complete(client: httpx.AsyncClient, prompt: str, *, attempts: int = 5):
payload = {"model": "gpt-fast", "messages": [{"role": "user", "content": prompt}]}
for attempt in range(attempts):
resp = await client.post("http://localhost:4000/chat/completions", json=payload)
if resp.status_code != 429:
resp.raise_for_status()
return resp.json()
wait = float(resp.headers.get("retry-after", 2 ** attempt))
await asyncio.sleep(wait + random.uniform(0, 0.5)) # jitter avoids sync retries
raise RuntimeError("shaped out: gateway rejected every attempt")
async def main():
sem = asyncio.Semaphore(8) # client-side bound: never exceed gateway slots
async with httpx.AsyncClient(timeout=60) as client:
async with sem:
return await complete(client, "Summarize incident #4210")
Three rules in that snippet. Read retry-after — LiteLLM returns the seconds until the current window resets. Add jitter, or every client retries at the same instant and re-trips the limit. And bound your own concurrency with a semaphore: the client should never offer more in-flight work than the gateway is configured to admit. That’s backpressure propagating all the way back to the producer.
Budgets: The Final Admission Control
Rate limits bound rate; budgets bound total. LiteLLM checks spend at every layer — key, user, team, global — and reserves the estimated cost of an in-flight request against the budget before dispatching it, reconciling with actual cost afterwards. Concurrency can’t slip past a reservation the way it can slip past a post-hoc check.
litellm_settings:
max_budget: 50.0 # $50 across the whole proxy
budget_duration: 1d # resets daily
Budgets require a database (they read spend from it), so don’t configure them on a DB-less deployment expecting enforcement. With fail_closed_budget_enforcement you can require a verifiable spend check before every admission, trading database load for a guarantee that a stale Redis counter never lets a key overspend.
Putting It All Together
Wire the pieces into one ingress policy:
- Every agent process points at the gateway, never at a provider directly.
- The gateway authenticates the virtual key, checks rpm/tpm/parallel limits, and either admits or returns
429 + retry-after. - Teams encode priority: interactive traffic has deep slots, batch traffic shallow ones.
- The router picks a deployment with spare capacity, retries with backoff, and fails over on cooldown.
- Clients bound their concurrency, honor
retry-after, jitter their retries, and shed their own low-priority work when their budget of attempts runs dry.
A load test at 200 requests/second against a rpm: 60 ceiling produces the output you want — deterministic rejection instead of melting:
admitted: 60 rejected_429: 140 p50_latency: 0.81s p99_latency: 1.02s errors: 0
Compare that with no ingress shaping: 100% of requests admitted, p99 latency 47 seconds, three upstream provider lockouts, and a retry storm. Same offered load, radically different system.
Conclusion and Next Steps
You’ve seen that backpressure at the LLM ingress is not one feature but a stack: bounded queues that reject rather than accumulate, hard rpm/tpm admission control, exact request counts versus reconciled token counts, priority classes for load shedding, budget reservations as the final ceiling, and clients that treat 429 as instructions rather than insults. LiteLLM gives you all of it in one config file because the gateway is the only component that sees every producer at once.
Homework: (1) put your own agent fleet behind the proxy and graph current_rpm versus queue depth — depth leads latency every time; (2) set max_parallel_requests on a team and watch requests fail fast with a clear 429 instead of queuing; (3) add a second proxy instance with Redis and verify the shared counters hold the limit at roughly N×1, not N×instances; (4) deliberately exhaust a small budget and confirm your client degrades to the free/local model rather than going dark.