If you have built even a small RAG application on Azure, you already know the language model is rarely the weak link. The retrieval step decides what the model actually sees, and if that step brings back irrelevant or low quality chunks, no amount of prompt engineering fixes the output on the other end. The Azure AI Search team ran a set of experiments comparing keyword search, vector search, hybrid search and semantic ranking, and the results give a clear, evidence backed answer on which combination works best for grounding generative AI applications.
This piece walks through what they tested, why hybrid retrieval combined with semantic ranking wins consistently, and what that means for how you should configure Azure AI Search in a production RAG pipeline.
Two Layers Behind Every Search Query
Azure AI Search structures every query into two layers, a pattern common in serious search systems. The first layer, called L1 or retrieval, scans the entire index, which can run into millions or even billions of documents, and returns a shortlist, typically around fifty candidates. The second layer, L2 or ranking, takes that shortlist and reorders it using a more computationally expensive model.
One thing worth remembering here: L2 can only reorder what L1 already found. If the ideal document never makes it into that shortlist of fifty, no amount of clever ranking downstream will recover it. That is why the choice of L1 mode matters as much as the ranker you put on top of it.
- Keyword uses classic full text search: content is broken into terms through language specific analysis, an inverted index is built, and the BM25 model scores the matches.
- Vector converts both documents and queries into embeddings and retrieves by nearest neighbour distance. The team used Azure OpenAI’s text-embedding-ada-002 model with cosine similarity throughout these tests.
- Hybrid runs keyword and vector retrieval side by side and fuses the two result sets using Reciprocal Rank Fusion, Azure AI Search’s current fusion algorithm.
On top of whichever L1 mode you pick, Azure AI Search offers a semantic ranker as the L2 layer. It runs the query and each candidate chunk through transformer models, adapted from the same models used in Bing, applying cross attention to produce a calibrated score between 0 and 4. A score of 0 means the chunk is essentially irrelevant, 4 means it is an excellent match. Because the scale stays consistent across indexes and queries, you can treat it as an actual relevance filter rather than just a sort key.
The Headline Number: Hybrid Plus Semantic Ranking Wins
Search Configuration Customer (NDCG@3) Beir (NDCG@10) Miracl (NDCG@10)
--------------------------------------------------------------------------------------
Keyword 40.6 40.6 49.6
Vector (Ada-002) 43.8 45.0 58.3
Hybrid (Keyword+Vector) 48.4 48.4 58.8
Hybrid + Semantic ranker 60.1 50.0 72.0Table 1. NDCG comparison across retrieval configurations, measured on customer datasets, the Beir academic benchmark, and the multilingual Miracl benchmark.
Look closely at where the biggest jump happens. On the customer datasets, adding semantic ranking on top of hybrid retrieval takes the score from 48.4 to 60.1, a jump of nearly 12 points. On Beir, the jump from hybrid to hybrid plus semantic ranker is much smaller, just 1.6 points.
This tells you something practical. Semantic ranking earns its cost most on messy, real world enterprise content, not on clean academic benchmark text. If your documents are noisy PDFs, scanned reports or inconsistent internal wikis, the ranker is doing real work. If your content already reads like a well curated benchmark dataset, you may see smaller gains and should measure on your own data before assuming the ranker is worth the extra latency and cost.
Why Hybrid Beats Either Method Alone Across Query Types
Query type Keyword Vector Hybrid Hybrid+Semantic
-------------------------------------------------------------------------
Concept seeking 39.0 45.8 46.3 59.6
Fact seeking 37.8 49.0 49.1 63.4
Exact snippet search 51.1 41.5 51.0 60.8
Web search-like 41.8 46.3 50.0 58.9
Keyword queries 79.2 11.7 61.0 66.9
Low query/doc overlap 23.0 36.1 35.9 49.1
Queries w/ misspellings 28.8 39.1 40.6 54.6
Long queries 42.7 41.6 48.1 59.4
Medium queries 38.1 44.7 46.7 59.9
Short queries 53.1 38.8 53.0 63.9Table 2. NDCG@3 comparison across ten different query types, from short keyword queries to long concept seeking questions.
One row deserves your attention: queries that are nothing but short identifier words, such as a product code or an API name. Keyword search scores 79.2 there, while pure vector search collapses to 11.7. This is a trap many teams fall into when they treat vector search as a universal upgrade over keyword search. Embeddings are good at conceptual similarity, not exact identifier matching. Hybrid retrieval recovers most of that lost ground, scoring 61.0 on the same query type, because its keyword component still catches the exact match even when the vector component misses it.
On the other end, queries with misspellings or low term overlap between the query and the document favour vector and hybrid heavily over pure keyword search, because embeddings tolerate typos and paraphrasing that a keyword index cannot. This is the core argument for hybrid over either method in isolation. Your users will type all of these query types in the same application, sometimes in the same session, and no single retrieval mode handles all of them well.
Chunking Strategy Changes Your Recall More Than You Would Expect
Chunking solves three problems for a RAG application. It splits long documents into passages short enough that several of them fit inside the model’s context window. It lets the most relevant part of a document surface first instead of being buried inside an otherwise average scoring whole-document vector. And it keeps each piece of text within the embedding model’s own token limit, which for Ada-002 is 8,192 tokens, though most deep embedding models cap out closer to 512.
Query type (Recall@50) Single vector/doc Chunked docs
----------------------------------------------------------------
Answer in long documents 28.2 45.7
Answer deep in a document 28.7 51.4Table 3. Recall@50 comparison between embedding a whole document as one vector versus chunking into 512 token pieces with 25 percent overlap.
The gap is largest exactly where you would expect it to hurt: queries whose answer sits deep inside a long document. Recall jumps from 28.7 to 51.4 once you chunk instead of embedding the whole document as one vector. If your corpus has long documents such as policy manuals or technical specifications, skipping chunking is not a shortcut, it is a real accuracy tax.
Chunk size (tokens/vector) Recall@50
------------------------------------------
512 42.4
1024 37.5
4096 36.4
8191 34.9Table 4. Recall@50 for different chunk sizes using the Ada-002 embedding model, vector retrieval only, no semantic ranking applied.
Smaller chunks retrieve better, and the pattern is close to linear. A 512 token chunk scores 42.4 Recall@50, while cramming 8,191 tokens, Ada-002’s full limit, into a single vector drops that to 34.9. An embedding model has to compress everything in the chunk into a fixed size vector, 1,536 dimensions in Ada-002’s case. Stuff too many unrelated ideas into one chunk and the vector becomes an average that represents none of them well. This is worth remembering the next time someone suggests embedding whole documents to save on chunking engineering effort. It will cost you retrieval quality.
Chunk boundary strategy Recall@50
----------------------------------------------------
512 tok, break at token boundary 40.9
512 tok, preserve sentence boundary 42.4
512 tok, 10% overlap 43.1
512 tok, 25% overlap 43.9Table 5. Recall@50 for different chunk boundary strategies, all using 512 token chunks with the Ada-002 embedding model.
Splitting on raw token boundaries and cutting sentences in half is the worst option in this comparison, scoring 40.9. Respecting sentence boundaries alone lifts that to 42.4, and adding 25 percent overlap between chunks takes it to 43.9. None of this is complicated engineering, but it is easy to skip when you are racing to get a RAG proof of concept working, and skipping it quietly costs you retrieval accuracy from day one.
Semantic Ranking Puts the Right Answer Where the LLM Will Actually See It
This point is easy to underestimate. Most production RAG pipelines do not pass the LLM 50 retrieved chunks, they pass 3 to 5, both to control token cost and to avoid diluting the model’s attention with irrelevant text. The chart above shows hybrid plus semantic ranking consistently finds the high quality chunk inside that narrow top 1 to 5 window more often than any other configuration, at every result set size tested. Retrieval quality at rank 50 does not help your application if it only reads the top 3.
Practical Takeaways for Your Own RAG Pipeline
A few things are worth carrying into your own architecture decisions. Enable hybrid retrieval by default rather than choosing between keyword and vector search. Your users will issue both kinds of queries in the same application, and hybrid handles both reasonably well without requiring you to detect query intent upfront.
Budget for semantic ranking specifically when your source content is messy or heterogeneous, think internal documents, scanned PDFs, mixed formatting, because that is where the evidence shows the largest lift. If your content is already clean and well structured, measure the gain on your own data before assuming it is worth the added latency, since semantic ranking adds a network hop and processing time to every query.
Chunk your documents. Do not embed entire long documents as a single vector to save engineering time, the recall numbers above show this is a real cost, not a minor rounding error. Aim for roughly 512 token chunks, respect sentence boundaries, and use around 25 percent overlap between chunks as a sensible starting point, then tune from your own evaluation data.
It is also worth being honest about the limits of this benchmark. The customer datasets, query sets, and GPT based scoring prompts used here are Microsoft’s own and are not something you can reproduce exactly on your side. The team documents their methodology in the appendix, but you should still validate hybrid plus semantic ranking against your own domain data before treating these numbers as guaranteed for your workload. A benchmark run on legal contracts or customer support tickets inside your organization could look quite different from academic search benchmarks or Microsoft’s customer sample.
How the Benchmarks Were Measured
The team scored results using three metrics. NDCG, Normalized Discounted Cumulative Gain, gives a score between 0 and 100 based on whether a retrieval system found the best results and put them in the right order. NDCG@10 covers the top 10 documents and was used for public benchmarks to stay consistent with prior published runs. NDCG@3 covers only the top 3 documents and was used for customer data, since that is closer to how many chunks a real RAG application actually reads. Recall@50 counts how many of the known good documents for a query show up somewhere in the top 50 retrieved results.
For scoring, the team used official labels and scoring libraries for the Beir and Miracl public benchmarks. For customer datasets, they used a GPT based scoring prompt that had been validated against a set of human reviewed ground truth labels. Queries came from a mix of real end user logs and GPT generated questions grounded in random snippets from each document index.
Leave a Reply