Structured Logging for Agent Decisions: Events, Inputs, and Outputs

Traditional logging captures what happened. Agent observability captures what was considered at each decision point. When a multi-step agent hallucinates a citation, traditional monitoring sees the error. Agent observability sees that “the summarization tool received a malformed context window at step three, which caused the downstream sub-agent to hallucinate.”

Why Traditional Logging Fails

An agent’s behavior is non-deterministic. The same prompt produces different tool call sequences depending on model temperature, retrieved context, or prior memory state. Traditional logging captures discrete events — CPU usage, request latency, error rates. Agent logs need to be structured around decisions, not events.

The Three-Surface Taxonomy (AgentTrace)

The AgentTrace framework (AAAI 2026 Workshop) introduces three logging surfaces:

Cognitive Surface — Internal reasoning: thought processes, planning steps, confidence scores, decision branches, intermediate reasoning chains.

Operational Surface — Explicit agent method calls, argument structures, return values, execution timing. Automatically intercepted via Python introspection.

Contextual Surface — External environment interactions: HTTP requests, SQL queries, cache/VectorDB operations. Stored as OpenTelemetry spans.

Each log event includes: UUID, surface type, trace ID, span ID, UTC timestamp, and event body.

OpenTelemetry for Agent Observability

The OpenTelemetry GenAI Semantic Conventions define standardized attributes:

gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.response.finish_reasons
gen_ai.agent.name
gen_ai.tool.call.id
gen_ai.tool.call.result

Agent-specific span conventions: create_agent, invoke_agent_client, invoke_workflow, plan, execute_tool.

Practical Instrumentation

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-system")

def run_agent(task):
    with tracer.start_as_current_span("agent_run") as span:
        span.set_attribute("agent.name", "research_agent")
        span.set_attribute("agent.version", "2.1.0")
        
        # Planning phase
        with tracer.start_as_current_span("plan"):
            plan = llm.plan(task)
            span.set_attribute("plan.steps", len(plan.steps))
        
        # Execution phase
        for step in plan.steps:
            with tracer.start_as_current_span(f"execute_{step.type}"):
                result = execute_step(step)
                span.set_attribute("step.tool", step.tool)
                span.set_attribute("step.confidence", result.confidence)

Best Practices

Log Decisions, Not Just Events

At each decision point, log: step name and type, policy version, model version, input fingerprint, output hash, and confidence score.

Input Fingerprinting and Output Hashing

import hashlib

def fingerprint(data):
    return hashlib.sha256(str(data).encode()).hexdigest()[:16]

# When you see divergent outputs for the same input fingerprint, flag it
input_hash = fingerprint({"query": user_query, "context": retrieved_docs})
output_hash = fingerprint(response)

Hierarchical Span-per-Tick Tracing

Each discrete reasoning step generates a distinct span. Spans nest hierarchically: parent trace for full run → child spans for LLM calls, tool invocations, memory operations, sub-agent handoffs.

Business Metadata from Day One

Tag every span with at least user_id and workflow_id. Adding these retroactively after a production incident is painful.

Intelligent Sampling

Log every decision for failed workflows, sample at 10-20% for successful ones. Never skip logging for failures. A typical agent workflow produces 20-50 structured log entries per run.

The Tools Landscape (2026)

Tool	Focus	Differentiator
Langfuse	LLM observability + evaluation	Open source, self-hostable, 10 observation types
Arize Phoenix	Model/agent monitoring	OTel-native, drift detection, embedding visualization
MLflow	End-to-end AI platform	One-line auto-tracing, framework agnostic
LangSmith	Trace collection + evaluation	Deep LangChain/LangGraph integration
AgentOps	Local-first observability	Passive hooks, time-travel debugging

Langfuse Example

from langfuse import observe

@observe()
def research_agent(query: str):
    # Automatically captures inputs, outputs, timings, errors
    docs = retrieve(query)
    analysis = analyze(docs)
    return synthesize(analysis)

Langfuse infers agent graphs automatically from observation timing and nesting. Supports full-text search across all trace inputs/outputs/metadata.

Multi-Agent Governance

Standardized telemetry across agents requires common semantic conventions. The OWASP Agent Observability Standard maps agent steps to OpenTelemetry spans with structured reasoning attributes (agent.thought, agent.reasoning) and agent/model metadata.

Azure AI Foundry best practices:

1. Benchmark-driven model selection
2. Continuous evaluation in development and production
3. CI/CD pipeline integration with auto-evaluation
4. AI red teaming before production
5. Monitoring with tracing, evaluations, and alerts

Sensitivity Handling

Hash, truncate, or redact sensitive data in span attributes. The Microsoft Agent Framework provides an EnableSensitiveData flag. Production recommendation: never log raw user inputs in spans — use fingerprints and truncated previews.

The Takeaway

Agent observability starts with structured decisions, not structured events. Adopt OpenTelemetry GenAI semantic conventions from day one, tag spans with business metadata, and choose a tool (Langfuse for open source, Arize for enterprise). Log every failed workflow fully, sample successful ones, and fingerprint inputs for anomaly detection.

💡 Start with @observe() from Langfuse — it captures inputs, outputs, timings, and errors with zero configuration.