Azure OpenAI On Your Data with Semantic Kernel

Every team building a chatbot on top of Azure OpenAI eventually hits the same wall. The model answers general questions well, but it knows nothing about your company’s own documents, policies, or product catalog. Azure OpenAI On Your Data is Microsoft’s answer to this problem, and when you pair it with Semantic Kernel, you get a grounded question-answering setup without writing your own retrieval pipeline by hand.

This article walks through the full setup: indexing a sample document in Azure AI Search through Azure OpenAI Studio, wiring that index into a Semantic Kernel console app, and comparing the answers before and after grounding is switched on. I have also added a few observations on when this approach makes sense in production and when you are better off building your own retrieval logic.

What On Your Data actually does

On Your Data is a chat extension built into Azure OpenAI. Instead of sending only your prompt to the model, it first queries an Azure AI Search index, pulls back the most relevant chunks of your documents, and stuffs them into the model’s context before generation. This is retrieval augmented generation, but Microsoft has packaged the retrieval half of it so you do not have to write the search-and-stitch code yourself.

The appeal is speed of setup. You point Azure OpenAI Studio at a storage account and a search service, upload your files, and the studio handles chunking, embedding, and indexing for you. For a proof of concept or an internal tool with a handful of documents, this saves real engineering time.

What you need before starting

Three Azure services are involved here, and each plays a distinct role in the pipeline.

• Azure OpenAI service, which hosts the chat model (GPT-3.5-Turbo or GPT-4) that actually answers questions.
• Azure AI Search service, which stores and indexes your data so it can be searched at query time.
• Azure Blob storage, which holds the raw data files. This one is optional for the feature to work, but Azure OpenAI Studio uses it as the staging area when you upload and index files through the portal.

If you already have a populated Azure AI Search index from an earlier project, you can skip the upload steps below and jump straight to the Semantic Kernel code, since the index is the only thing the application actually talks to.

Indexing a sample document through Azure OpenAI Studio

Open the Chat Playground in Azure OpenAI Studio, switch to the Add your data tab, and click Add a data source. This is the entry point for connecting a search index to your deployment without touching any code.

The next screen asks for your subscription, the Azure Blob storage account, and the Azure AI Search service, along with a name for the index that will be created. Make sure Add vector search to this search resource is turned on. Once enabled, a dropdown appears asking which embedding model to use, since the studio needs to convert your files into vectors before they can be indexed for similarity search.

For a test run, a single PDF is enough. The sample used here comes from the Azure AI Search sample data repository and describes two employee health insurance plans, Northwind Health Plus and Northwind Standard, along with what each plan covers. This is intentionally content the base model has never seen, so any correct answer about it later proves the grounding is working and not just the model guessing.

The wizard then asks which search type to enable. Vector search compares embeddings for semantic similarity, while Hybrid search combines vector similarity with traditional keyword search. Hybrid is usually the safer default in production because it catches exact term matches that pure vector search sometimes misses, but Vector alone works fine for this walkthrough.

Next comes the authentication method for the search service. If your Azure AI Search resource has RBAC enabled, System assigned managed identity is the cleaner choice since there is no key to rotate or leak. This walkthrough uses API key authentication instead, mainly because it is quicker to set up when you are just testing the feature end to end.

Reviewing and confirming the configuration kicks off the actual indexing job, which takes a couple of minutes for a single small file and longer for a real document set. Once it finishes, open the Azure AI Search resource in the portal, go to Search management, Indexes, and confirm the new index exists. The Search explorer tool inside that page lets you run a quick query directly against the index, which is the fastest way to confirm your data actually made it in before you write a single line of application code.

Wiring the index into a Semantic Kernel app

With the index confirmed, create a new console project and add the Semantic Kernel package.

dotnet new console -n azure-semantic-kernel-quickstart
dotnet add package Microsoft.SemanticKernel

This gives you a bare console app with the Semantic Kernel SDK referenced. Nothing runs yet since there is no code wired up, but it confirms the package restores correctly before you start writing logic.

Six pieces of information are needed to connect everything: the Azure OpenAI endpoint and key (found under Resource Management, Keys and Endpoint in the portal), the deployment name, and the Azure AI Search endpoint, key, and index name (found under Overview and Settings, Keys on the search resource, plus Search management for the index name). Set these as environment variables rather than hardcoding them.

export AZURE_OPENAI_ENDPOINT=REPLACE_WITH_YOUR_AOAI_ENDPOINT_VALUE_HERE
export AZURE_OPENAI_API_KEY=REPLACE_WITH_YOUR_AOAI_KEY_VALUE_HERE
export AZURE_OPENAI_DEPLOYMENT_NAME=REPLACE_WITH_YOUR_AOAI_DEPLOYMENT_VALUE_HERE
export AZURE_AI_SEARCH_ENDPOINT=REPLACE_WITH_YOUR_AZURE_SEARCH_ENDPOINT_VALUE_HERE
export AZURE_AI_SEARCH_API_KEY=REPLACE_WITH_YOUR_AZURE_SEARCH_API_KEY_VALUE_HERE
export AZURE_AI_SEARCH_INDEX=REPLACE_WITH_YOUR_INDEX_NAME_HERE

Keeping keys out of source code is not just good hygiene, it also means you can swap between a local test index and a production one just by changing environment variables, without touching the application code at all. In a real deployment you would pull these from Azure Key Vault or App Configuration rather than shell exports, but exports are fine for a local proof of concept.

Next, build the Kernel and register Azure OpenAI as the chat completion service.

var kernel = Kernel.CreateBuilder()
    .AddAzureOpenAIChatCompletion(
        deploymentName: Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT_NAME"),
        endpoint: Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT"),
        apiKey: Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY"))
    .Build();

This is a plain Semantic Kernel setup with no data grounding attached yet. At this point the Kernel behaves exactly like a normal chat completion client, so it is worth testing this baseline before adding any complexity.

Confirming the model knows nothing about your data

Ask a question that only makes sense if the model has access to the indexed document.

var result = await kernel.InvokePromptAsync("What are my available health plans?");

Since the Kernel has no connection to the search index yet, the model has no way to know about Northwind Health Plus or Northwind Standard. The response comes back as something like: “I’m an AI language model, and I don’t have access to your personal information or healthcare options.” This confirms the baseline model is not grounded in your data, which is exactly what you want to see before wiring in the search connection, otherwise you cannot be sure a later correct answer isn’t just a lucky guess.

Grounding the response with Azure AI Search

To connect the search index, create an AzureSearchChatExtensionConfiguration with the search endpoint, authentication, and index name, then attach it to the execution settings passed into the prompt call.

var azureSearchExtensionConfiguration = new AzureSearchChatExtensionConfiguration
{
    SearchEndpoint = new Uri(Environment.GetEnvironmentVariable("AZURE_AI_SEARCH_ENDPOINT")),
    Authentication = new OnYourDataApiKeyAuthenticationOptions(Environment.GetEnvironmentVariable("AZURE_AI_SEARCH_API_KEY")),
    IndexName = Environment.GetEnvironmentVariable("AZURE_AI_SEARCH_INDEX")
};
 
var chatExtensionsOptions = new AzureChatExtensionsOptions { Extensions = { azureSearchExtensionConfiguration } };
var executionSettings = new OpenAIPromptExecutionSettings { AzureChatExtensionsOptions = chatExtensionsOptions };
 
var result = await kernel.InvokePromptAsync("What are my available health plans?", new(executionSettings));

This is the same prompt as before, but the execution settings now carry the Azure AI Search connection. Under the hood, Azure OpenAI takes your question, runs a search against the index, retrieves the matching chunks from the health plans PDF, and feeds them to the model alongside your original question before it generates a response. The response now reads along these lines: “You have two available health plans through Contoso Electronics: Northwind Health Plus and Northwind Standard [doc1]. Northwind Health Plus is a comprehensive plan that provides coverage for medical, vision, and dental services… Northwind Standard is a basic plan that provides coverage for medical, vision, and dental services, as well as preventive care services and prescription drug coverage…”

The [doc1] citation marker at the end is worth paying attention to. It is the signal that the answer was actually pulled from a specific indexed document rather than generated from the model’s general training data. If you ever see a plausible-sounding answer without this citation marker, treat it with suspicion, since it usually means the extension did not find a good match and the model filled the gap on its own.

Where this approach fits, and where it does not

On Your Data is a strong fit when you need to stand up a grounded chatbot quickly and your data does not need heavy custom processing before indexing. It is also convenient when the team building the chatbot is not the same team that owns the search infrastructure, since the studio wizard removes a lot of coordination overhead.

The trade-off is control. You do not get to customize the chunking strategy, tune the retrieval ranking, or inject your own re-ranking logic between search and generation, since Azure OpenAI is managing that internally. If your documents have unusual structure, such as large tables or nested sections that need custom chunking rules, or if you need tight control over which chunks get sent to the model for cost or accuracy reasons, a hand-rolled RAG pipeline using the Azure AI Search plugin directly in Semantic Kernel gives you that control at the cost of more code to maintain.

Cost is another practical factor. On Your Data adds a search call on every single request, and depending on your traffic pattern, a hand-built pipeline can be tuned to cache embeddings, batch searches, or skip retrieval entirely for questions that clearly do not need it. None of that is available to you when the studio is managing the pipeline. For an internal tool with light traffic, this rarely matters. For a customer-facing product with real scale, it is worth prototyping both approaches before committing.