Skip to content
Blog

Backpressure for Agent Ingress: Shaping Load at the LLM Gateway

How to apply real backpressure at the ingress of an agent system with LiteLLM: bounded queues, rpm/tpm limits, load shedding, budget-aware admission, and correct 429 handling.

Published on • November 16, 2026

AI Assistant

Your agent works. Then someone points it at a queue of 10,000 documents, every document spawns a sub-agent, and every sub-agent fires five model calls. Within seconds your gateway is saturated, latency climbs from 800ms to 90 seconds, retries pile on retries, and the whole system collapses under its own enthusiasm. The provider’s 429s are not the problem — they are the symptom you failed to design for.

In this post, you will learn how to shape load at the ingress of an agent system using LiteLLM as an LLM gateway: bounded queues and why queue depth is your best signal, token-bucket versus sliding-window limiting, per-key and per-model rpm/tpm limits, priority classes and load shedding, correct client-side 429 handling, and budgets as the final admission control. You’ll ship a runnable proxy config and a Python client that behaves under load.

Prerequisites

  • Python 3.10+ and pip install 'litellm[proxy]' openai httpx
  • A provider API key (OpenAI, Anthropic, Azure, or any of the 100+ LiteLLM supports)
  • Familiarity with HTTP status codes and basic concurrency (asyncio.Semaphore)
  • Optional but recommended: Redis, if you run more than one proxy instance

Where Backpressure Belongs: The Ingress

Backpressure is the discipline of telling producers “slow down” before work accumulates unboundedly. In an agent system the natural choke point is the LLM gateway — every model call funnels through it, so it’s the one place where admission control is both possible and effective.

  agent fleet                 ingress (backpressure here)              providers
┌─────────────┐        ┌───────────────────────────────────┐        ┌──────────┐
│ orchestrator│  HTTP  │  LiteLLM proxy                    │        │ OpenAI   │
│  └ subagents├───────►│  1. auth: virtual key lookup      │───────►│ Azure    │
│  └ subagents│        │  2. admission: rpm/tpm/parallel   │  rpm   │ Anthropic│
│  └ subagents│        │     └ bounded queue + shed        │  tpm   │ Bedrock  │
└─────────────┘        │  3. router: pick deployment       │        └──────────┘
     ▲                 │  4. budget check (reservation)    │
     │ 429 + retry-after│  5. post-request: usage + spend  │
     └─────────────────┴───────────────────────────────────┘

Note what the diagram forbids: unbounded queues. A queue with no limit isn’t resilience, it’s deferred failure — requests age in memory until their deadlines are meaningless. Every queue in the path gets a maximum depth, and when it fills, new work is rejected or shed, not parked.

Queues, Tokens, and Windows: The Theory

Two limiting algorithms dominate, and knowing which one you’re using changes how you interpret its errors.

Token bucket. N tokens refill at a constant rate; each request consumes tokens (of request count, or of tokens). Bursts are allowed up to the bucket size. It’s simple, memory-cheap, and naturally tolerates spikey traffic — the right default when your users are interactive.

Sliding window (fixed-window counters). Count requests in the current minute, reset when the minute rolls over. LiteLLM’s limiter works this way: counters like current_rpm and current_tpm are checked and incremented per minute window, and the response carries retry-after seconds until that window resets. The trade-off is exactness at the boundary — a fixed window can admit 2× the limit across a window edge, which is why enforcement is described as exact for RPM (checked before the call) but best-effort for TPM (tokens are only known after the response lands).

That last point deserves emphasis: request counts are admission-controlled; token counts are reconciled after the fact. LiteLLM reserves tokens ahead of the call and corrects against actual usage afterwards, which is why TPM can slightly overrun under concurrency.

Finally, queue depth. When a gateway does queue — or when your client-side queue grows — depth is the earliest, most actionable signal you have. Rising depth with flat throughput means you’re saturated, not slow. Alert on depth crossing a threshold long before latency does.

Configuring LiteLLM as the Shaping Layer

LiteLLM’s proxy takes a YAML config. This one declares two deployments of the same model with hard rpm/tpm ceilings, a concurrency cap, and Redis-backed counters so multiple proxy instances agree on the limit:

model_list:
  - model_name: gpt-fast
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
    rpm: 60            # 60 requests per minute
    tpm: 900000        # 900k tokens per minute
    max_parallel_requests: 8

  - model_name: gpt-fast
    litellm_params:
      model: azure/gpt-4o-mini-west
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE

router_settings:
  routing_strategy: simple-shuffle      # weighted by rpm/tpm
  num_retries: 2
  cooldown_time: 30
  optional_pre_call_checks:
    - enforce_model_rate_limits         # rpm/tpm become hard limits
  redis_host: os.environ/REDIS_HOST
  redis_port: 6379

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Run it with litellm --config config.yaml and point any OpenAI-compatible client at http://localhost:4000. Two behaviors matter for backpressure:

  • Without enforce_model_rate_limits, rpm/tpm only guide routing. With it, requests over the ceiling are blocked before reaching the provider and return 429 with a retry-after header — genuine admission control rather than hopeful scheduling.
  • When every slot governed by max_parallel_requests is busy, a new request fails immediately with a 429 naming the deployment. There is no invisible wait queue; earlier LiteLLM versions silently queued and produced mysterious 200-second latencies, which is precisely the anti-pattern this design rejects.

For multi-instance deployments, Redis shares the counters; while Redis is unreachable each instance falls back to in-memory counters (fail-open), and fail_closed_rate_limit_enforcement flips that to a hard ceiling if your limits must hold during an outage.

Priority Classes and Load Shedding

Not all agent traffic deserves equal treatment. LiteLLM models this with teams: a key inherits its team’s max_parallel_requests, tpm_limit, and rpm_limit, so you can carve the ingress into priority classes.

curl -X POST 'http://localhost:4000/team/new' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"team_id": "interactive", "max_parallel_requests": 8, "rpm_limit": 120, "tpm_limit": 2000000}'

curl -X POST 'http://localhost:4000/team/new' \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"team_id": "batch", "max_parallel_requests": 2, "rpm_limit": 20}'

Interactive chat gets eight slots; the nightly batch gets two. When the provider tightens its quota, you shed load from the bottom: batch jobs absorb the 429s while interactive latency stays flat. You can go further by routing budget-exhausted traffic to explicitly free models — set input_cost_per_token: 0 and output_cost_per_token: 0 on a local deployment and LiteLLM skips budget checks for it, giving you a degraded-but-alive path instead of a dead end.

Handling 429 on the Client

The gateway’s 429 is a contract: not admitted, try later, here’s when. Honor it. A client that retries on a fixed 100ms schedule is adding fuel to the fire.

import asyncio, random, httpx

async def complete(client: httpx.AsyncClient, prompt: str, *, attempts: int = 5):
    payload = {"model": "gpt-fast", "messages": [{"role": "user", "content": prompt}]}
    for attempt in range(attempts):
        resp = await client.post("http://localhost:4000/chat/completions", json=payload)
        if resp.status_code != 429:
            resp.raise_for_status()
            return resp.json()
        wait = float(resp.headers.get("retry-after", 2 ** attempt))
        await asyncio.sleep(wait + random.uniform(0, 0.5))   # jitter avoids sync retries
    raise RuntimeError("shaped out: gateway rejected every attempt")

async def main():
    sem = asyncio.Semaphore(8)          # client-side bound: never exceed gateway slots
    async with httpx.AsyncClient(timeout=60) as client:
        async with sem:
            return await complete(client, "Summarize incident #4210")

Three rules in that snippet. Read retry-after — LiteLLM returns the seconds until the current window resets. Add jitter, or every client retries at the same instant and re-trips the limit. And bound your own concurrency with a semaphore: the client should never offer more in-flight work than the gateway is configured to admit. That’s backpressure propagating all the way back to the producer.

Budgets: The Final Admission Control

Rate limits bound rate; budgets bound total. LiteLLM checks spend at every layer — key, user, team, global — and reserves the estimated cost of an in-flight request against the budget before dispatching it, reconciling with actual cost afterwards. Concurrency can’t slip past a reservation the way it can slip past a post-hoc check.

litellm_settings:
  max_budget: 50.0            # $50 across the whole proxy
  budget_duration: 1d          # resets daily

Budgets require a database (they read spend from it), so don’t configure them on a DB-less deployment expecting enforcement. With fail_closed_budget_enforcement you can require a verifiable spend check before every admission, trading database load for a guarantee that a stale Redis counter never lets a key overspend.

Putting It All Together

Wire the pieces into one ingress policy:

  1. Every agent process points at the gateway, never at a provider directly.
  2. The gateway authenticates the virtual key, checks rpm/tpm/parallel limits, and either admits or returns 429 + retry-after.
  3. Teams encode priority: interactive traffic has deep slots, batch traffic shallow ones.
  4. The router picks a deployment with spare capacity, retries with backoff, and fails over on cooldown.
  5. Clients bound their concurrency, honor retry-after, jitter their retries, and shed their own low-priority work when their budget of attempts runs dry.

A load test at 200 requests/second against a rpm: 60 ceiling produces the output you want — deterministic rejection instead of melting:

admitted: 60   rejected_429: 140   p50_latency: 0.81s   p99_latency: 1.02s   errors: 0

Compare that with no ingress shaping: 100% of requests admitted, p99 latency 47 seconds, three upstream provider lockouts, and a retry storm. Same offered load, radically different system.

Conclusion and Next Steps

You’ve seen that backpressure at the LLM ingress is not one feature but a stack: bounded queues that reject rather than accumulate, hard rpm/tpm admission control, exact request counts versus reconciled token counts, priority classes for load shedding, budget reservations as the final ceiling, and clients that treat 429 as instructions rather than insults. LiteLLM gives you all of it in one config file because the gateway is the only component that sees every producer at once.

Homework: (1) put your own agent fleet behind the proxy and graph current_rpm versus queue depth — depth leads latency every time; (2) set max_parallel_requests on a team and watch requests fail fast with a clear 429 instead of queuing; (3) add a second proxy instance with Redis and verify the shared counters hold the limit at roughly N×1, not N×instances; (4) deliberately exhaust a small budget and confirm your client degrades to the free/local model rather than going dark.