Timeouts and Deadlines: Human Fallback When Agents Stall

An agent that hangs is worse than one that fails. A failure raises an error you can catch and retry; a hang holds a worker, a session, and a user’s patience hostage while every dashboard stays green — because “in progress” is indistinguishable from “making progress” unless you built the distinction in.

The fix isn’t one timeout but layered deadlines, a definition of stall beyond wall-clock, and a degradation path ending with a human holding something actionable.

Three deadlines, not one

Conflating these is the most common design error:

1. Per-call timeout — one LLM or tool invocation must return within N seconds. Kills hung HTTP requests.
2. Per-step budget — one agent iteration (reason → act → observe) has a ceiling: max steps, max tokens, max time.
3. Per-run deadline — the whole task gets an absolute deadline. Nothing resets it.

Only #3 guarantees termination. A loop of 50 calls each taking 10 seconds will happily consume 8 minutes under a 10-second per-call timeout.

import asyncio, time

class RunDeadline:
    def __init__(self, seconds: float):
        self.total = seconds
        self.start = time.monotonic()

    @property
    def remaining(self) -> float:
        return self.total - (time.monotonic() - self.start)

    async def guard(self, coro, cap: float | None = None):
        budget = min(self.remaining, cap or self.remaining)
        if budget <= 0:
            raise TimeoutError("run deadline exceeded")
        return await asyncio.wait_for(coro, timeout=budget)

Note the min(...): each call’s timeout is clamped to the remaining run budget, so the last call can’t overrun the deadline. Make the run deadline absolute from the start — resetting it on every iteration is how “10-minute tasks” become hour-long incidents.

Gateway-level timeouts

If your calls go through an LLM gateway, put the boring reliability machinery there instead of hand-rolling it per service. LiteLLM — an open source AI gateway with a Rust core and Python SDK, calling 100+ providers in OpenAI format — exposes the distinctions we care about directly:

• Global: litellm_settings.request_timeout (or REQUEST_TIMEOUT) is the proxy-wide default when no router or deployment timeout is set. On native /v1/responses and /v1/messages streams it bounds each wait for the next chunk, so a stalled upstream errors instead of hanging.
• Router-level: Router(timeout=30) caps the entire call.
• Per-deployment: timeout caps the complete response; stream_timeout caps the wait for the first token — the knob that catches providers that accept your request and then go silent.

litellm_settings:
  request_timeout: 60        # proxy-wide default
  num_retries: 2             # per model group
  allowed_fails: 3           # cooldown after repeated failures
  cooldown_time: 30

model_list:
  - model_name: fast-agent
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      timeout: 30            # whole response
      stream_timeout: 10     # time to first token
      keepalive_seconds: 15  # SSE : ping frames on idle streams

keepalive_seconds is the subtle one: idle streams get killed by load balancers that think the connection is dead. The gateway emits SSE comment frames (: ping) every N seconds — invisible to clients, but enough to keep proxies from closing the connection. It’s operator-only by default (a client enabling heartbeats at will could hold a connection open indefinitely), clamped to 1–300 seconds, and settable proxy-wide with sse_keepalive_ping_interval_seconds.

Timeouts can also be set per request, which is how you give a “deep research” call a longer leash than a chat turn:

response = router.completion(
    model="fast-agent",
    messages=[{"role": "user", "content": "summarize the incident"}],
    timeout=120,
)

Detecting stalls in your own loop

Gateway timeouts catch hung calls, not a converged-looking loop that keeps stepping forever. You need an explicit stall detector:

def is_stall(step, state) -> str | None:
    if step.count >= 25:
        return "step budget exhausted"
    if step.inputs_hash == state.prev_hash:
        state.repeat_count += 1
        if state.repeat_count >= 3:
            return "same action repeated 3x"
    else:
        state.repeat_count = 0
    if deadline.remaining <= 0:
        return "run deadline exceeded"
    return None

Three signals cover most real stalls: repetition (same tool call + same args, state unchanged), oscillation (A → B → A → B, detectable by hashing a sliding window of recent states), and non-convergence (no reduction in unresolved work items over N steps). Hash the observable state — the arguments you’d pass to the tool — not the model’s prose, which paraphrases itself into false novelty every turn.

Falling back: model first, human second

Degradation is a ladder, and each rung should be automatic until the last one:

1. Retry the same call (bounded — num_retries in LiteLLM does this per model group).
2. Fall back to another model group. LiteLLM’s fallbacks config does exactly this: after num_retries failures, fail over to a healthy backup. Typed variants cover content-policy and context-window errors, default_fallbacks is a catch-all, cooldowns (allowed_fails / cooldown_time) take a sick deployment out of rotation, and per-request disable_fallbacks: true returns the raw error.

litellm_settings:
  fallbacks: [{"fast-agent": ["backup-model"]}]
  default_fallbacks: ["backup-model"]

3. Degrade the output — answer from what’s already known, marked as partial, instead of blocking on the unreachable tool.
4. Escalate to a human — when retries, fallbacks, and the run deadline are all exhausted.

That last rung is a product surface, not a raise. The handoff payload should contain the original task, every step with tool name / args / result summary, the stall reason, the deadline that fired, and a checkpoint of mutable state so the human doesn’t start from zero. Then pick a channel matching urgency: a queue item for review, an alert for a stuck pipeline, or an in-UI banner.

Design the resume path too: an escalation resolvable only by restarting the run trains humans to avoid it.

Test the failure path deliberately

The failure path is code you write once and discover broken during an incident. LiteLLM’s "mock_timeout": true on /chat/completions makes a request behave as if the provider timed out, so you can exercise retry/fallback logic without waiting for a real outage.

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "fast-agent",
    "messages": [{"role": "user", "content": "hi"}],
    "mock_timeout": true
  }'

Wire it into CI as a chaos test: a forced timeout must produce a human-escalation record within the run deadline, not a hung job.

Observability you’ll actually alert on

• Time-to-first-token and total call time per tool, overlaid with the configured timeout. Rising p99 against a flat timeout is your early warning.
• Stall rate — runs terminated by stall detection ÷ total runs. This tracks the agent getting worse separately from providers getting slow.
• Escalation rate and time-to-human — rising escalations with flat provider metrics means your agent logic regressed, not your vendors.
• Deadline headroom — runs finishing in the last 10% of budget are one flaky tool call from an incident.

Pitfalls and gotchas

• A gateway timeout doesn’t stop your loop. If your code catches the timeout and immediately issues another call, you’ve built a retry storm, not a timeout. Enforce the run deadline yourself.
• timeout and stream_timeout are different things. One caps the complete response, the other the wait for the first chunk. Setting only the first lets a provider hang you for its full duration before a single token appears.
• Keepalive pings change your error surface. Once the proxy has sent an early ping, the status line is committed — a later failure arrives as an SSE error frame under HTTP 200, not a 5xx, and x-litellm-* headers are absent. Your client must handle in-stream errors.
• Retrying non-idempotent tools double-spends. A timeout means “I don’t know if it happened”, not “it didn’t happen”. Make tool calls idempotent (idempotency keys) or verify state before retrying.
• Timeouts that are too tight cascade. If the gateway timeout is longer than the caller’s, you hold connections the caller already abandoned. Set deadlines inner < outer, and make the run deadline the tightest of all.
• Fallbacks multiply token spend during incidents — each is an extra attempt against a second provider. Bound them (num_retries, cooldown_time) and watch spend during failovers.
• Never escalate with just an error string. “Timeout after 30s” is unactionable. Include the step, arguments, and state — otherwise the human restarts blind and hits the same wall.
• An unbounded “human fallback” is a silent queue. Alert on escalation age; an item nobody picks up is a hang with extra steps.

Sources

• BerriAI/litellm — open source AI gateway
• LiteLLM docs — Timeouts (global, per-deployment, stream, per-request, keepalive, mock_timeout)
• LiteLLM docs — Fallbacks (provider failover, retries, cooldowns, disable_fallbacks)