LlamaIndex Agents vs. a General-Purpose SDK: When to Reach for Which

“Should I use LlamaIndex or a general-purpose agent SDK?” is the wrong question to ask in the abstract. The right question is: where does your leverage come from — plumbing your data into the model’s context, or controlling exactly what the model is allowed to do? LlamaIndex agents are optimized for the first problem. SDKs like PydanticAI and the OpenAI Agents SDK are optimized for the second. This post walks through what each side actually gives you, straight from the docs, and where the boundary sits.

What LlamaIndex agents give you

Per the LlamaIndex agent docs, an agent is “a semi-autonomous piece of software powered by an LLM that is given a task and executes a series of steps towards solving that task.” Crucially, its tools “can be anything from arbitrary functions up to full LlamaIndex query engines.” That single sentence is the architectural thesis: in LlamaIndex, a RAG pipeline is not a special mode — it is just a tool the agent can call.

A managed tool-calling loop, with several strategies

LlamaIndex ships pre-built agent workflows so you don’t write the loop yourself. FunctionAgent uses the LLM provider’s native function/tool calling; ReActAgent and CodeActAgent use different prompting strategies; AgentWorkflow composes multiple agents that can hand control to each other. The docs spell out the loop explicitly: the agent takes the latest message plus chat history, sends tool schemas and history over the API, and either responds directly or returns tool calls — the tool results go back into chat history and the agent is invoked again.

Query engines as first-class tools

This is the part no thin SDK replicates. The tools guide documents QueryEngineTool, which wraps an existing query engine as a tool, and FunctionTool, which converts any Python function into a tool with auto-inferred schema. Since LlamaIndex agent classes inherit from BaseQueryEngine, a tool can even wrap another agent. A retrieval agent is then a few lines:

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import QueryEngineTool

query_engine = index.as_query_engine()
tool = QueryEngineTool.from_defaults(
    query_engine, name="docs_search", description="Search the product docs"
)

agent = FunctionAgent(tools=[tool], llm=llm, system_prompt="Answer using docs only.")
response = await agent.run("How do I rotate API keys?")

Document and index tooling that a bare SDK doesn’t have

Beyond the loop, LlamaIndex brings the data plane: data connectors, indexes and vector stores, and query engines as composable parts of the framework. Two utility tools show the flavor:

• OnDemandLoaderTool turns any LlamaIndex data loader into a tool that, in a single tool call, loads data from the source, indexes it (for instance with a vector store), and queries it on demand.
• LoadAndSearchToolSpec wraps any tool so it exposes a load tool and a search tool — the output of the underlying API call gets indexed so large responses don’t overflow the context window.

There are also community ToolSpecs — bundles of tools around one service, like Gmail — and return_direct=True on a tool, which ends the agent’s reasoning loop and returns the tool’s response verbatim (useful to cut latency on final-answer retrieval steps). Memory is handled too: agents default to a ChatMemoryBuffer you can customize with a token limit.

Workflows when you need more control

When the pre-built loop isn’t enough, LlamaIndex has Workflows — an event-driven model where a step receives an event, does work, and returns another event that triggers the next step. Branches are ordinary if statements returning different event types; loops are steps returning events handled by earlier steps. Event types are Pydantic objects, and the workflow’s event graph is validated before it runs (start and stop events present, produced events have consumers, no accidental dead ends). The docs even cover human-in-the-loop steps and durable workflows that checkpoint context and resume after a restart.

What a general-purpose SDK gives you

Typed contracts end to end (PydanticAI)

PydanticAI is “typed end to end”: you give an agent an output_type and every run returns a validated Pydantic model; tools receive a typed RunContext carrying your dependencies; tool arguments are validated before your code runs, and validation errors are fed back to the LLM for a retry. The model is a string you can swap ('openai:gpt-...', 'anthropic:...'), so no flagship feature is locked to one vendor. When plain control flow isn’t enough, Pydantic Graph brings the same typing to graph-based workflows, and durable execution runs the same agent inside Temporal, DBOS, or Prefect with human-in-the-loop approval built in.

Few primitives, Python-first orchestration (OpenAI Agents SDK)

The OpenAI Agents SDK advertises “very few abstractions”: agents (LLMs with instructions and tools), agents-as-tools/handoffs for delegation, and guardrails that validate inputs and outputs in parallel with agent execution. Function tools get automatic schema generation with Pydantic-powered validation, sessions provide a persistent memory layer, and tracing is built in for debugging and evaluation. Non-OpenAI models are supported through multi-provider adapters.

The common thread: neither SDK knows what a chunker, an embedding index, or a query engine is. They give you a typed, portable runtime and expect your retrieval to arrive as ordinary tools.

Decision matrix

Workload	Reach for
Data-retrieval-heavy RAG copilot over documents, APIs, or databases	LlamaIndex agents — QueryEngineTool, loaders, and indexes do the heavy lifting
Business workflow with strict typed tools, typed outputs, and human-in-the-loop approval	PydanticAI or the OpenAI Agents SDK — validation and approval are first-class
Multi-agent orchestration with explicit state and durable execution	Graph frameworks (Pydantic Graph, LangGraph) or LlamaIndex Workflows — see our agent SDK landscape deep dive for the alternatives
Provider-portable core logic with minimal abstraction	Thin SDK (PydanticAI, OpenAI Agents SDK)
ingestion-and-retrieval over hard documents (scans, tables, slide decks)	LlamaIndex (+ LlamaParse)

Signs you’ve outgrown LlamaIndex agents

• Your agent’s value is in typed contracts and compliance, not retrieval: every tool call must be validated against a strict schema, and every output must round-trip through a Pydantic model with approval gates.
• You need durable, multi-day execution on an engine you already operate (Temporal, Prefect), not just a checkpointed workflow run.
• The retrieval is trivial (one vector store, one SQL query) while the orchestration is complex — routing, delegation, retries with typed state. LlamaIndex Workflows cover a lot of this, but at that point you’re maintaining a graph framework anyway; a typed SDK may be the leaner home.

Signs you’re over-engineering with a bare SDK

• You’re hand-rolling document loaders, chunking, embedding, indexing, and citation attribution — all solved problems in LlamaIndex.
• Your “tools” list is slowly reinventing OnDemandLoaderTool and LoadAndSearchToolSpec with worse context management.
• Answer quality depends on retrieving the right passage from a large private corpus. That is LlamaIndex’s home turf, and fighting it means writing RAG infrastructure you now own.

The hybrid pattern: LlamaIndex for retrieval, your SDK for control

You don’t have to pick one. The cleanest production setup treats LlamaIndex as a retrieval library inside a typed agent:

1. Build your index and query engine with LlamaIndex (connectors, parsing, embedding, reranking).
2. Wrap the query engine as one function tool — either via QueryEngineTool inside a FunctionAgent, or by calling the retriever from a plain function registered on your PydanticAI or OpenAI Agents SDK agent.
3. Own the orchestration — state, approvals, provider choice, tracing — in the SDK, where the typing and portability live.

The inverse also works: use LlamaIndex purely for ingestion and indexing, and let your SDK agent call the served index over HTTP. Either way, each tool does the job it was designed for.

Closing

The trade-off boils down to leverage: LlamaIndex agents buy you the data plane (connectors, indexes, query engines, retrieval-aware tools) plus a managed loop; a general-purpose SDK buys you typed contracts, explicit control flow, and provider portability. Start with LlamaIndex when retrieval is the product, start with an SDK when the workflow is the product, and combine them the moment both are.

Primary sources:

• LlamaIndex: Building an agent
• LlamaIndex: Agents module guide
• LlamaIndex: Tools
• LlamaIndex: Workflows
• PydanticAI
• OpenAI Agents SDK