Skip to content
Blog

Self-Healing Workflows: Detecting and Recovering from Stuck Agents

Autonomous agents stall — loops, silent failures, unresponsive tools. Here's how to instrument for stall detection and build recovery paths that get workflows moving again.

Published on • October 28, 2026

AI Assistant

Agent workflows fail in ways traditional services don’t. A web server either responds or throws; an agent can spend forty minutes in a tool loop, re-read the same file eleven times, or silently produce a plausible-looking answer derived from nothing. These are stalls — the workflow isn’t down, it just isn’t making progress. And because stalled workflows still burn tokens, they’re more expensive than crashes.

Self-healing is the discipline of detecting stalls automatically and recovering without a human in the loop. It has two halves: instrumentation that notices, and recovery paths that act.

Taxonomy of Stalls

You can’t detect what you haven’t classified. The common stall patterns:

1. Degenerate loops

The agent repeats the same tool call with the same arguments, or oscillates between two states (read file A → read file B → read file A…). The tell is low unique-action ratio: many tool calls, few distinct ones.

2. Silent tool failure

A tool returns an error the model doesn’t recognize as fatal, so the agent retries it identically forever — or worse, treats the empty error result as “nothing found” and concludes wrongly. No exception is raised anywhere; the workflow “succeeds” with bad output.

3. Context drowning

The agent’s context fills with retrieval results and tool transcripts until the original instruction is buried. Behavior degrades to answering from noise. The signal is monotonically growing context with decreasing task-relevant content.

4. Unresponsive dependencies

A tool calls an API that hangs. Unlike a crash, a hang produces no error to react to — only time passing.

5. Plan divergence

The agent’s stated plan and its actual actions drift apart: it said it would verify step 3, but steps 3-9 are all “more research.” Task-level progress flattens while activity continues.

Instrumentation: Making Stalls Visible

You cannot heal what you cannot see. Instrumentation is the foundation; OpenTelemetry’s GenAI semantic conventions give you the vocabulary — spans for model calls and tool executions, attributes for token usage, model identity, and tool names.

Metrics that catch stalls

  • Tool-call uniqueness ratio — distinct calls / total calls over a window. Below a threshold (say 0.5), you’re looping.
  • Repeated-argument count — identical (tool, arguments) pairs beyond a bound is a hard stall signal.
  • Step-to-step latency — spikes indicate unresponsive tools; a p99 that exceeds your deadline budget means timeout enforcement is missing.
  • Task-progress slope — plan items completed per unit time. Flat slope + nonzero spend = divergence.
  • Context growth rate — tokens per step; near-linear growth with a fixed task is the drowning pattern.

Structured logging for decisions

Log agent decisions as structured events, not free text: action taken, rationale (a short model-generated summary), inputs, outputs. This is what makes post-hoc diagnosis possible — and it’s the substrate for the “should we have intervened?” analysis that tunes your detectors.

Trace the hierarchy

Distributed tracing applies directly: an orchestrator span, child spans per sub-agent, grandchildren per tool call. A stall is then a span that’s too long, a subtree that repeats, or a leaf that never resolves. Visualization matters — “debugging loops and dead ends” is fundamentally a spatial problem that trace waterfalls solve.

Detection: Rules and Models

With signals in place, detection has two layers:

Threshold rules (fast, cheap)

First line of defense. Examples:

if identical(tool, args) count > 3 within 5 min → stall
if progress_slope == 0 for 10 min AND tokens_spent > budget/4 → stall
if step_latency > tool_timeout → kill step, escalate

Rules are deterministic, explainable, and run at near-zero cost. Tune them from incident post-mortems.

LLM judges (catch the subtle ones)

Some stalls are semantic: the agent is “working” but answering a different question than it was asked. A lightweight judge model can periodically assess: Given the task and the recent trajectory, is the agent still making meaningful progress? Bounded to a few percent of total spend, judges catch plan divergence and goal substitution that thresholds miss.

Recovery Paths: From Detection to Healing

Detection without recovery just pages someone faster. Recovery escalates through a ladder:

Level 1: Nudge

Inject a corrective message into the agent’s context: “You have called read_file on the same path 4 times. If the file is unchanged, proceed using what you have or report the blocker.” Surprisingly effective for degenerate loops — models often exit loops when the loop is pointed out to them.

Level 2: Reset with summary

When context is drowned, don’t retry in the same context. Terminate the run, generate a compact summary of completed work, and spawn a fresh agent with the summary plus the original task. This is the agent-native version of “turn it off and on again” — and it works for the same reason.

Level 3: Checkpoint resume

For frameworks with durable execution (LangGraph checkpointers, Temporal-style workflows), kill the stalled run and resume from the last checkpoint with a modified policy — a different model, a restricted toolset, an injected constraint. Checkpointing turns recovery from “start over” into “rewind and retry differently.”

Level 4: Degraded completion

Sometimes the right outcome is a partial result with explicit caveats: “Completed steps 1-3; step 4 blocked by tool timeout; attached findings so far.” Shipping an honest partial beats an infinite loop or a fabricated whole.

Level 5: Human escalation

The terminal case. Hand off with full context: the task, the trajectory, the stall diagnosis, and what was tried. Escalation isn’t failure — it’s the system knowing its own limits, which is exactly what self-healing is for.

Building the Loop

agent run
  ├─ spans/metrics/logs → collector
  ├─ detectors (rules + judge) ──stall?─→ recovery ladder
  │                                     ├─ nudge → back into run
  │                                     ├─ reset-with-summary → new run
  │                                     ├─ checkpoint-resume → new run
  │                                     ├─ degraded completion → done
  │                                     └─ human escalation → done
  └─ every recovery recorded → detector tuning corpus

Two properties make this self-healing rather than self-stopping:

  1. Recovery actions are themselves observable. Did the nudge work? Did the reset produce a better trajectory? If recovery attempts aren’t traced, you’re flying blind twice over.
  2. Every stall feeds the tuning corpus. Detectors that never improve will either miss novel stalls or drown teams in false positives — both of which get the system turned off.

Testing Recovery Deliberately

Don’t wait for organic stalls. Inject them:

  • Fault injection — make a tool hang, return empty results, or 500 intermittently (chaos engineering, agent edition).
  • Adversarial tasks — give the agent a task designed to induce looping (an impossible constraint, a deliberately contradictory instruction) and verify the detectors fire and recovery completes.
  • Replay tests — record a stalled trajectory, replay it against the detector stack, assert the stall is flagged. This runs in CI at zero model cost.

The goal: your recovery ladder has been exercised more times in testing than in production.

Conclusion

Stalled agents are worse than crashed agents — they consume budget while appearing alive. Self-healing workflows close the loop: instrument with GenAI semantic conventions, measure loop ratios and progress slopes, detect with cheap rules plus bounded LLM judges, and recover through an escalating ladder — nudge, reset, checkpoint-resume, degrade, escalate. Test the whole machinery with injected faults before production does it for you. Agents will still stall; the difference is your system notices in seconds and gets moving again on its own.

Further Reading