Most retrieval augmented generation demos stop at a single vector index and one call to an LLM. That setup looks impressive in a five minute walkthrough, but it falls apart once real users start typing layered, ambiguous questions against messy enterprise data spread across SharePoint, Blob Storage, OneLake, and half a dozen line of business systems. A weak retrieval layer produces a weak agent, no matter how capable the underlying model is.
At Microsoft Ignite 2025, the session “Better grounding for AI apps with Azure AI Search knowledge retrieval” (BRK193) walked through how to design that retrieval layer properly. This article follows that session and explains, in plain terms, how keyword search, vector search, Reciprocal Rank Fusion, and cross encoder reranking come together inside Azure AI Search, and how Knowledge Bases and Foundry IQ turn that stack into something an agent can actually reason with rather than just query.
Why every agent needs a retrieval layer
The session uses a fictional retailer called Zava to make the point concrete. Zava runs a Shopper Agent that finds products, an Interior Design Agent that finds design images, an Inventory Agent that checks stock, and an HR Agent that answers policy questions. Every one of these agents is only as good as the data it can pull in at query time.
Each of these agents depends on domain specific knowledge: catalog data, design assets, inventory feeds, HR documents. Without a dependable retrieval engine sitting between the agent and that data, you get hallucinations, missing context, and agents that confidently answer questions they have no business answering. This is exactly the gap that retrieval augmented generation is meant to close.
RAG in its simplest form, and where it breaks
The basic RAG pattern is easy to describe. A user asks a question, the system sends it to a search index, the index returns a handful of relevant document chunks, those chunks plus the original question go to an LLM, and the LLM writes an answer grounded in what it was given. Azure’s own guidance on RAG with AI Search follows exactly this pattern: index your data, retrieve relevant passages, ground the answer in that context.
This works well for narrow, single intent questions against a single source. It starts to break down the moment you have multiple sources to search, multi step questions such as comparing last year’s policy against a new one, or a requirement for strong grounding, citations, and security trimming that respects who is actually allowed to see what. The fix is not a bigger index or a smarter prompt. The fix is to stop treating RAG as one query plus one index, and instead treat retrieval as a multi step, agentic pipeline.
Hybrid search: the actual retrieval backbone
Azure AI Search’s retrieval stack is built from four layers stacked on top of each other: keyword search, vector search, fusion, and reranking. Each layer covers a weakness in the one before it, and skipping any of them shows up as a specific, predictable failure mode in production.
Keyword search runs on a classic inverted index and uses BM25, which remains one of the strongest full text ranking algorithms available. It is excellent at exact phrase and precise term matches, for example a query for a “25 foot hose”, but it has no concept of meaning. Ask it for something semantically related using different words and it simply will not find it.
Vector search fixes that blind spot by embedding both the query and the documents into vectors and measuring similarity through cosine or dot product distance. Azure AI Search uses HNSW, a graph based approximate nearest neighbor algorithm, to keep this fast even at large index sizes. A query like “water plants efficiently without waste” can now surface a sprinkler system even though the word sprinkler never appears in the query.
Neither keyword nor vector search wins on its own, so hybrid retrieval runs both and merges the two ranked lists using Reciprocal Rank Fusion. RRF scores each document using its rank position in both result sets rather than raw relevance scores, which avoids the awkward problem of comparing a BM25 score against a cosine similarity score on two completely different scales.
score(doc) = 1 / (k + rank_keyword) + 1 / (k + rank_vector)
# k = 60 in Azure AI Search's default configurationA document that ranks highly in either list gets a meaningful boost, and a document that ranks highly in both lists rises to the top. The constant k dampens the effect of rank differences further down the list, so a document sitting at rank 2 does not dominate one sitting at rank 3 nearly as much as intuition might suggest. This is standard practice in information retrieval and is not something Azure invented, but having it built into the search service saves you from hand rolling your own fusion logic.
Fusion alone still leaves noise near the top of the list, which is where the fourth layer comes in. A cross encoder reranking model scores each candidate document against the original query directly, using a model trained on human relevance judgments rather than a generic distance metric. Microsoft’s own benchmarks show hybrid search plus reranking beating both pure vector and pure keyword search across a wide range of query types, and in practice this is the layer that pushes the truly best answer to position one instead of position four.
Put together, this four layer stack, keyword, vector, fusion, and reranking, is the foundation that everything else in Knowledge Bases and agentic retrieval sits on top of. If this layer is weak, no amount of clever orchestration above it will fix the answers your agent gives.
When hybrid search alone is not enough
Even a well tuned hybrid search stack breaks down against three classes of queries that show up constantly in real usage. The first is a single query that actually contains multiple questions, such as asking what paint suits a bathroom and what the price range of good quality options looks like. That needs two separate search intents resolved and merged, not one search call.
The second is a chained query, where the answer to the first part determines what to search for in the second part, for example asking how to paint a house efficiently and then asking for a shopping list of matching products and prices. The third is a query that needs both internal catalog data and external knowledge that was never indexed at all, such as asking which sander suits table corners and when to replace the sanding pad, where the replacement guidance is general knowledge rather than a Zava product fact.
Past this point you are no longer working with a single prompt, single search call, single answer pattern. You need something that behaves like an agent: it decomposes the query, decides which sources to search, retries when the first pass comes back thin, and merges everything into one coherent answer. That is precisely the job Knowledge Bases in Azure AI Search are built to do.
Knowledge Bases: the agentic retrieval engine
A Knowledge Base wraps three components around the hybrid search stack described above. Query planning uses an LLM to break a conversation into distinct search intents, and it is what recognizes when a single user question is actually two or three questions stitched together or requires sequential reasoning. Knowledge source selection then decides which indexes, lakes, SharePoint sites, web sources, or MCP endpoints are worth querying for each intent, instead of blindly hitting every source for every query.
The third component, results merging and answer synthesis, collects everything that came back, merges it into one ranked set, and optionally drafts an answer with citations plus an activity log describing exactly what the engine did and why. If the engine judges the results insufficient, it can loop back, re run query planning using the earlier results as context, refine or branch the queries, and pull in additional sources such as a web search through Bing. Think of a Knowledge Base as RAG with a brain attached to it: instead of hand coding all of this branching logic into your application, you delegate it to a managed engine running inside Azure AI Search.
Microsoft’s documentation on agentic retrieval and knowledge sources walks through creating the index, defining knowledge sources, and wiring a Knowledge Base into an application or agent, and is worth reading end to end before you build your first one.
Indexed versus remote knowledge sources
A Knowledge Base can pull from two very different classes of sources, and picking the wrong one for a given scenario is a common early mistake. Indexed knowledge sources ingest data into Azure AI Search as a proper search index, drawing from Blob Storage, OneLake in Fabric, or SharePoint document libraries through an indexer plus skillset pipeline. That pipeline runs on a schedule with change tracking for additions, updates, and deletions, and it can apply built in skills such as Content Understanding, SplitSkill for chunking, and EmbeddingSkill for vectorization. Parsing can run in a minimal mode, which is free and fine for plain text, or a standard mode that uses Content Understanding to enrich documents with things like media descriptions and structured fields.
Remote knowledge sources take the opposite approach: the data never gets ingested at all and stays exactly where it lives, with the Knowledge Base calling out to the source at query time. Bing web search, SharePoint accessed through a remote semantic index, and MCP endpoints, currently in private preview, all fall into this category. Remote sources still honor filters and security, since you can pass user tokens, domain filters, or site IDs to keep the results scoped to what the requesting user is actually allowed to see.
The session illustrates this with a SharePoint scenario worth remembering, because it is a trap teams fall into in production. A Sales employee asks what an executive decision was, and the Knowledge Base queries the remote SharePoint semantic index with an authorization header carrying the user’s token. If that user’s access is not perfectly aligned with what got semantically indexed, the query can silently come back with zero results, even though the document genuinely exists and the user genuinely has access to it in SharePoint itself. Ingesting the same SharePoint content through an indexer and skillset instead lets you respect user and group IDs directly at query time and return the relevant content reliably.
The practical rule of thumb from the session is straightforward. Use indexed sources when you want full control over chunking, enrichment, and ranking quality. Use remote sources when you need live data, a SaaS system, or external web content that would be wasteful or impossible to keep fully synchronized. Most production Knowledge Bases end up combining both in the same configuration, letting the agentic engine decide which to pull from for a given query.
Reasoning effort: minimal, low, and medium
A Knowledge Base exposes a reasoning effort setting with three levels, and this is effectively a latency, quality, and cost slider that you tune per call rather than per deployment.
Minimal effort takes the search intents you provide as is, sends all of them to all configured sources, runs hybrid search on each, and merges the results. It skips query planning, source selection, web sources, answer synthesis, and any iterative retry. This is the right choice for simple question and answer scenarios where you already know exactly which sources matter, and for cost sensitive, low latency paths where an extra planning round trip is not worth the delay.
Low effort adds real query planning across the full conversation rather than static, pre supplied intents, plus source selection so only the sources relevant to each intent get queried. It also adds an optional answer synthesis step and an activity log, but it still skips iterative retrieval, so there is no second pass if the first round of results comes back weak. Most production chat and agent experiences that do not involve genuinely hard multi part queries land here, since it gives materially better routing and answers without the overhead of multiple retrieval rounds.
Medium effort is where the engine becomes genuinely agentic. A semantic classifier decides whether the current context is actually enough to answer the question, and if it is not, the engine runs a second retrieval pass with more sophisticated merging and relevance scoring. This gives the highest quality and the most agentic behavior of the three tiers, at the cost of more model tokens, higher latency, and higher spend. It fits complex, multi part, or chained queries, and it is the right choice for agents where a wrong answer is genuinely expensive, think financial, healthcare, or internal engineering tooling.
A sensible default in practice is to reserve minimal effort for internal tools and deterministic flows where you already control the query shape, and to use low or medium for anything customer facing. It is worth resisting the urge to default every Knowledge Base call to medium effort just because it produces the best individual answer. That habit quietly inflates both latency and token spend across an application, and most queries genuinely do not need a second retrieval pass to answer correctly.
How the pieces fit together: architecture and Foundry IQ
Pulling everything together, the reference architecture has three layers. The data plane handles ingestion and indexing: content sits in Blob, OneLake, or SharePoint, indexers and skillsets pull it in, chunk it, enrich it, and push it into an Azure AI Search index, generating embeddings through Azure OpenAI or another embedding model along the way. The Knowledge Base configuration layer then points at one or more indexed sources such as a product catalog or a policy library, optionally adds remote sources such as Bing or a SharePoint semantic index, and lets you set the reasoning effort per call.
On top of that sits the application and agent layer. A web app, a line of business system, or a Microsoft Foundry Agent sends the conversation to the Knowledge Base, which runs agentic retrieval and returns merged results with scores and citations, an optional activity log, and optionally a drafted final answer. This is where Foundry IQ enters the picture: Microsoft positions it as the knowledge layer for agents, and it lets an agent running inside Azure AI Foundry call a Knowledge Base, often over MCP, as one tool among several whenever it needs factual, grounded context rather than a general purpose answer from the model alone. The agent then folds that context into its plan and can chain it with other tools or actions as needed.
This architecture lines up with Microsoft’s existing RAG and hybrid search guidance for Azure AI Search and Azure OpenAI, but it replaces hand coded retrieval logic scattered across an application with one explicit, managed agentic retrieval layer that every agent in the organization can share.
Where to start building this
The session’s speakers pointed to a few concrete starting points rather than leaving this purely conceptual. The GPT-RAG reference repository covers general RAG patterns on Azure end to end, and the RAG Chat evaluator repository gives you a way to actually measure recall and precision for your own Azure AI Search configuration instead of guessing whether hybrid search or reranking made a real difference.
- RAG with Azure AI Search: learn.microsoft.com/azure/search/retrieval-augmented-generation-overview
- Hybrid search and reranking: learn.microsoft.com/azure/search/hybrid-search-ranking
- Knowledge Bases and agentic retrieval: learn.microsoft.com/azure/search/agentic-retrieval-how-to-create-index
- Knowledge sources for agentic retrieval: learn.microsoft.com/azure/search/agentic-knowledge-source-how-to-search-index
If you are already running a RAG system on Azure AI Search today, the practical next step is not to rebuild it from scratch. Model your existing data as a mix of indexed and remote knowledge sources, stand up a Knowledge Base with a reasoning effort that matches the actual complexity of your queries rather than the highest setting available, and integrate it as a tool your agents call rather than logic baked into a single application. That is the difference between a one off RAG demo and a retrieval layer your organization’s agents can keep reusing as new use cases show up.
Leave a Reply