Agentic RAG: retrieval that the agent controls
Classic RAG retrieves once and hopes for the best. Agentic RAG turns retrieval into a tool the agent plans, repeats and checks.
By Diego Bianqui · Updated
What RAG is
Retrieval-augmented generation (RAG) gives a model access to knowledge it wasn't trained on — your policies, products, contracts, tickets — by retrieving relevant passages at query time and placing them in the context before the model answers. It reduces hallucination, keeps answers current without retraining, and makes it possible to cite sources.
- Ingest: collect documents and keep them in sync with their sources.
- Chunk: split them into passages that make sense on their own.
- Index: embed the chunks for semantic search, and usually index keywords and metadata too.
- Retrieve: find the passages most relevant to the query.
- Augment: put the best passages into the model's context, labelled as source data.
- Generate: answer from those passages, ideally with citations.
Where classic RAG breaks
- One shot: a single retrieval with the user's raw question, even when the question is vague, compound or needs several steps.
- One source: everything goes to one vector index, even when the answer lives in a database, an API or a different document store.
- No self-check: irrelevant or contradictory passages go straight into the answer.
- No permissions: the index returns passages the user was never allowed to see.
What makes RAG agentic
In agentic RAG, retrieval stops being a fixed pre-processing step and becomes a set of tools the agent decides how to use. The agent can rewrite or decompose the question, choose which source to search, look at what came back, judge whether it is enough, and search again — before it answers or acts.
| Aspect | Classic RAG | Agentic RAG |
|---|---|---|
| Query | The user's question as typed | Rewritten, expanded or split into sub-questions |
| Sources | One index | Routed across vector search, keyword search, SQL, APIs and documents |
| Iterations | One retrieval | As many as needed, within a budget |
| Quality check | None | The agent grades relevance and looks for gaps or conflicts |
| Output | An answer | An answer, a decision or an action — grounded and cited |
The flexibility has a price: more model calls, more latency and more ways to go wrong. Agentic RAG needs the same discipline as any agent — budgets, tracing and evaluation from the harness and AgentOps.
Design decisions that matter most
- Chunking that respects structure: sections, tables and clauses, not arbitrary character counts.
- Hybrid retrieval: semantic search plus keyword search, so exact terms like product codes and clause numbers are found.
- Reranking: a second pass that orders candidates by real relevance before they reach the context.
- Metadata filters: date, product, region, document type — often more effective than a better embedding model.
- Permission-aware retrieval: filter by what the requesting user may see, at query time, not after generation.
- Citations: every claim traceable to a passage, so users and auditors can check it.
- Retrieval as tools: expose sources through clear tool definitions, for example via MCP.
Security: retrieved text is untrusted input
Anything a retriever returns can carry instructions: a poisoned wiki page, a malicious email, a manipulated document. Treat retrieved passages as data, never as instructions, and never let them trigger actions without validation. The OWASP Agentic Top 10 lists this as agent goal hijack (ASI01) and memory and context poisoning (ASI06) — see the guide. Leaking documents across permission boundaries is the other big RAG risk, and it is solved in the retrieval layer, not the prompt.
Measure retrieval and generation separately
When a RAG answer is wrong, the cause is either retrieval (the right passage wasn't found) or generation (it was found but misused). Evaluate both: context precision and context recall for retrieval, faithfulness and response relevancy for the answer. Libraries such as RAGAS implement these metrics — see evaluating agents and RAG.
Related guides
- Evaluating AI agents and RAG: evals, metrics and RAGAS
- Context engineering: what the model sees, and when
- AI agent protocols: MCP, A2A, AG-UI, A2UI, AP2 and UCP
Sources
- Ragas documentation, Available metrics (external site)
- OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications for 2026 (external site)
Working on agents in production? Get in touch.