Announcing Vector Search in Azure Cognitive Search Public Preview

In July 2023, Microsoft opened public preview access to vector search inside Azure Cognitive Search. The service was renamed Azure AI Search a few months later, but the capability launched here is the same one still powering most Retrieval Augmented Generation setups on Azure today. For anyone building on top of GPT-family models at the time, this was the missing piece that let you ground a language model in your own data instead of relying only on what the model already knew from training.

This article walks through what vector search actually does, how it sits alongside the keyword search you already know from Cognitive Search, and where hybrid retrieval fits into the picture. I have added a few production notes based on working with both the preview and the general availability release that followed, since a fair bit changed between July 2023 and GA.

What vector search solves that keyword search cannot

Traditional search in Cognitive Search matches tokens. If a document contains the word car and someone searches for automobile, a plain keyword index will not connect the two unless you have gone through the trouble of building a synonym map. Vector search works on a different principle. It converts both your content and the incoming query into numerical vectors called embeddings, then finds the nearest vectors by distance instead of matching words.

Azure OpenAI’s text-embedding-ada-002 model was the embedding model most teams reached for during this preview. You run each document chunk through the embeddings endpoint, store the resulting vector alongside the document in your search index, and compare it against the vector of any incoming query at search time. Because the comparison happens in vector space, a query about automobile maintenance can retrieve a document about car servicing even though the two share almost no vocabulary. This is not limited to text either. Azure AI Vision could convert images into the same kind of vector representation, which is how the platform supported image to image and text to image search alongside plain document retrieval.

Where a vector field fits inside an index you already have

One point worth clarifying, since it trips up people new to this feature: vector search did not replace the existing Cognitive Search index structure. It added a new field type that you could mix into an index you were already using for filters, facets, and regular text fields. Indexers pulling from Blob Storage, Azure SQL, or Cosmos DB kept working exactly as before. You simply added a vector field next to your existing fields and pointed a background process at it to populate the embeddings.

A vector field declaration in the preview schema looked roughly like the sample below. This is not lifted from the announcement itself, since the post was a high level overview rather than a code walkthrough, but it reflects the field structure that shipped with the preview and is documented in the accompanying GitHub sample.

{
  "name": "contentVector",
  "type": "Collection(Edm.Single)",
  "dimensions": 1536,
  "vectorSearchConfiguration": "my-vector-config",
  "searchable": true,
  "retrievable": true
}
 
"vectorSearch": {
  "algorithmConfigurations": [
    {
      "name": "my-vector-config",
      "kind": "hnsw"
    }
  ]
}

A few things matter in that definition. The dimensions value has to match whatever your embedding model outputs, which is 1536 for text-embedding-ada-002, so mixing embedding models within one field without reindexing will break silently or return poor results. The algorithm kind of hnsw refers to Hierarchical Navigable Small World graphs, an approximate nearest neighbor method that trades a small amount of recall for a large gain in query speed once your index grows past a few hundred thousand vectors. Exhaustive KNN was also available as an alternative for smaller indexes where exact results mattered more than shaving milliseconds off latency, and it is worth reaching for that option when your dataset is small enough that the performance difference does not matter.

How this plugs into a RAG application

Retrieval Augmented Generation, or RAG, is the pattern that made this preview relevant to nearly every generative AI project running on Azure that year. Instead of fine tuning a model on your own documents, which is expensive and quickly goes stale, you retrieve the most relevant chunks of your data at query time and pass them to the model as context alongside the user’s question. The model then generates its answer grounded in that retrieved content rather than purely from what it memorized during training.

Vector search is the retrieval half of that pattern. A user’s question gets embedded the same way your documents were embedded, the nearest document chunks are pulled from the index, and those chunks get stitched into the prompt sent to the LLM. This is why the timing of this preview mattered so much. Without a retrieval mechanism that understood semantic similarity rather than exact keyword overlap, RAG applications would miss a large share of relevant content whenever users phrased questions differently from how the source documents were written.

Why hybrid retrieval beats vector search on its own

Pure vector search is not automatically better than keyword search, and treating it that way is a common mistake I have seen teams make after this preview shipped. Vector similarity is excellent at catching paraphrases and conceptual matches, but it can miss exact terms that matter a great deal in enterprise search, things like product codes, error numbers, or acronyms that carry precise meaning rather than fuzzy semantic meaning. Keyword search is exactly the opposite: precise on exact terms, blind to paraphrasing.

Hybrid retrieval runs both search methods against the same query and merges the ranked results, which is why it consistently outperforms either method used alone in real world testing. Azure Cognitive Search supported this by letting you issue a single query that carried both a text search term and a vector, with the engine combining relevance scores from both paths before returning results. If you are building anything user facing, hybrid should be your default starting point rather than an optimization you add later.

The preview extended cleanly into capabilities teams were already using, which is part of why adoption moved quickly:

• Search enabled, chat based applications built on Azure OpenAI, where retrieved chunks feed directly into the prompt
• Text to image and image to image search powered by Azure AI Vision embeddings
• Fast retrieval across large, heterogeneous datasets pulled in through existing indexers from Blob Storage, Azure SQL, and Cosmos DB

What changed between this preview and general availability

Vector fields are heavier than they look on paper. A 1536 dimension float array per chunk adds up fast once you are indexing tens of thousands of documents, and I have seen teams get an unpleasant surprise on their Azure Cognitive Search bill after moving from a small proof of concept to a production-sized index without recalculating storage and replica costs first. Plan your service tier around vector storage from the start rather than treating it as an afterthought once keyword search was already sized.

Chunking strategy matters more than most teams expect going in. Chunk too large and the embedding averages out the specific detail a user was searching for, which hurts recall. Chunk too small and you lose surrounding context that the LLM needs to generate a coherent answer, and you also multiply your storage cost since every chunk carries its own full vector. There is no universal chunk size that works well across document types, so this is worth testing against your own content rather than copying a number from a blog post, including this one.

Being a public preview also meant the API surface was not final. Field names, algorithm configuration options, and query syntax shifted between this July 2023 preview and the GA release later that year when the service was renamed Azure AI Search. Anyone who built directly against the preview API had to revisit that code before going to production, and semantic ranking, which layers a re-ranking model on top of hybrid results, arrived only after this initial preview and is worth evaluating separately if relevance quality is a concern.

Where to start

Microsoft published a working sample alongside this preview that walks through creating a vector index, generating embeddings with Azure OpenAI, and running hybrid queries end to end, which is a faster starting point than piecing the flow together from documentation alone. The announcement also referenced Schroders, a financial services firm, using this capability alongside Azure OpenAI for multi modal search inside their internal AI solution, which is a reasonable real world example of the RAG pattern applied outside a demo environment.