Retry Policies and Fallbacks for Invalid Model Responses

Large Language Models (LLMs) are probabilistic engines. Even with structured output enforcement, models occasionally produce truncated JSON, hallucinate fields, or fail safety checks. In enterprise agent pipelines, an uncaught model validation error will crash execution and result in poor user experience.

To achieve production reliability, agent architectures need stateful retry policies, exponential backoffs, and automated fallback chains that route requests to backup models when primary inferences fail.

Architectural Patterns for Model Reliability

When an LLM produces an invalid response, there are three primary recovery strategies:

1. Immediate Reflection & Retry: Feed the validation error message back into the model prompt so it can correct its mistake in a follow-up turn.
2. Model Fallback Chain: Switch execution from the primary model (e.g., a fast lightweight model) to a more capable reasoning model (e.g., Gemini 3 Pro) after consecutive failures.
3. Deterministic Default Fallback: Return a safe pre-computed default state or human-in-the-loop signal if all retries and fallbacks are exhausted.

Implementing Stateful Retries in LangGraph

LangGraph’s graph-based execution model makes it straightforward to design stateful validation loops.

from typing import TypedDict, Annotated, Optional
from langgraph.graph import StateGraph, END
from pydantic import BaseModel, Field, ValidationError

class OutputSchema(BaseModel):
    action: str = Field(..., description="Action to execute")
    confidence: float = Field(..., ge=0.0, le=1.0)
    parameters: dict

class AgentState(TypedDict):
    input_text: str
    response_raw: Optional[str]
    parsed_output: Optional[OutputSchema]
    retry_count: int
    error_message: Optional[str]

def generate_node(state: AgentState):
    # Simulate LLM call with system prompt including error feedback if present
    prompt = f"User Request: {state['input_text']}"
    if state["error_message"]:
        prompt += f"\n\nPrevious attempt failed with error: {state['error_message']}. Please fix your response format."
    
    # In practice, call LLM model here
    raw_response = '{"action": "search", "confidence": 0.95, "parameters": {"query": "AI Agents"}}'
    return {"response_raw": raw_response, "retry_count": state["retry_count"] + 1}

def validate_node(state: AgentState):
    try:
        data = OutputSchema.model_validate_json(state["response_raw"])
        return {"parsed_output": data, "error_message": None}
    except ValidationError as err:
        return {"error_message": str(err.errors()), "parsed_output": None}

def router_condition(state: AgentState) -> str:
    if state["parsed_output"] is not None:
        return "success"
    if state["retry_count"] >= 3:
        return "fallback"
    return "retry"

Graph Wiring and Fallback Route

workflow = StateGraph(AgentState)

workflow.add_node("generate", generate_node)
workflow.add_node("validate", validate_node)

workflow.set_entry_point("generate")
workflow.add_edge("generate", "validate")

workflow.add_conditional_edges(
    "validate",
    router_condition,
    {
        "success": END,
        "retry": "generate",
        "fallback": END # Routes to fallback handler node or graceful error state
    }
)

app = workflow.compile()

Best Practices for Fallback Chains

• Cap Max Retries: Never allow open-ended retry loops. Limit in-context reflection retries to 2 or 3 attempts to prevent infinite billing loops.
• Vary Temperature: Lower the model temperature (e.g., set temperature=0.0) during retry turns to increase deterministic format compliance.
• Log Validation Telemetry: Instrument every validation failure with detailed traces to monitor schema compliance trends across model releases.

For additional graph orchestration patterns and state persistence strategies, check out the official LangGraph Documentation.