Tag based image search has a basic problem. You tag a photo as “sunset” and if someone searches for “orange sky over water” the system finds nothing, even though the photo is an exact match. Vector image search fixes this by comparing the actual visual content of images instead of relying on whatever labels a human happened to attach.
This article walks through a reference solution published on Microsoft’s Apps on Azure blog by eladt, a former Microsoft employee. It combines Azure AI Vision, Azure AI Search, Azure OpenAI, and Azure Functions to build a working image search pipeline, backed by a public sample repository called azure-ai-vision-search. I have gone through the architecture and the source code, and along with explaining how the pieces fit, I want to flag a few places where you should pause before taking this straight into production.
What you need before starting
The sample assumes you already have a handful of Azure services provisioned. Nothing here is exotic, but getting the combination right matters more than any single resource.
- An Azure subscription with permission to create resources
- An Azure Storage account to hold the source images
- An Azure AI Search service, any tier and any region
- An Azure OpenAI resource with a chat model deployed (the sample uses gpt-35-turbo)
- An Azure Functions app, Python based, ideally on a Premium plan since image processing can run long
- Azure CLI, VS Code, and Postman for testing the HTTP endpoints
How the pieces fit together
The high level flow is straightforward once you separate it from the security plumbing around it. A client sends a search query, an Azure Function refines that query using Azure OpenAI, converts it into a vector using Azure AI Vision, and runs a vector similarity search against an Azure AI Search index. On the ingestion side, the same Function exposes a second endpoint that Azure AI Search calls during indexing to convert each stored image into a vector.
Two things in this diagram are worth calling out for anyone planning a real deployment. First, every hop is drawn going through a private endpoint, and the diagram explicitly notes that all operations are role based. Second, there is a WAF and a firewall sitting in front of the Function App, which tells you this was designed as an internet facing service, not an internal tool. Keep this in mind because, as you will see further down, the actual configuration in the sample repository does not fully match this picture.
Setting up the Azure AI Search index, indexer, and skillset
Azure AI Search needs three things configured before it can serve vector search: an index that defines the schema, an indexer that pulls data from a source such as Blob Storage, and a skillset that defines any enrichment steps applied to the data before it lands in the index. The general shape of this pipeline looks like this.
The sample repository ships the JSON definitions for all three pieces (vector-image-index-db.json, vector-image-indexer.json, and vector-image-skillset.json), and the article walks through uploading each one via the Azure portal.
1. The index
The index JSON defines the fields, data types, and vector configuration for the search index. You upload this under the Indexes blade using the Add index (JSON) option.
2. The indexer
The indexer connects the index to a data source, in this case the Blob Storage container holding the images, and controls how often it runs and how it handles failures.
3. The skillset
The skillset is where the custom vectorize Function gets wired into the pipeline as a web API skill, so that every image the indexer picks up gets sent to your Function for embedding generation before it is written to the index.
One practical note here. Uploading these JSON files through the portal is fine for a quick proof of concept, but I would not do this for anything that has to survive past a demo. Portal edits are not tracked anywhere, so if a teammate tweaks the skillset six months from now, nobody knows what changed or why. Define these as Bicep or Terraform resources and deploy them through your CI pipeline instead, the same way you would treat any other infrastructure.
Configuring the Azure Function
The Function app needs a set of environment variables to talk to Azure OpenAI, Azure AI Vision, and Azure AI Search. These are normally stored in local.settings.json for local development and pushed into the Function App configuration blade for deployed environments.
export AZURE_OPENAI_API_KEY=<Your Azure OpenAI API Key>
export AZURE_OPENAI_ENDPOINT=<Your Azure OpenAI Endpoint>
export OPEN_AI_MODEL=gpt-35-turbo
export API_VERSION=2024-02-01
export AI_VISION_ENDPOINT=<Your Azure Vision Endpoint>
export AI_VISION_API_KEY=<Your Azure Vision API Key>
export AI_SEARCH_SERVICE_ENDPOINT=<Your Azure Search Service Endpoint>
export AZURE_SEARCH_ADMIN_KEY=<Your Azure Search Admin Key>
export AI_SEARCH_INDEX_NAME=<Your Azure Search Index Name>
export ACCOUNT_KEY=<Your Account Key>These map directly to the four services in play: the AZURE_OPENAI_* pair authenticates against your chat model deployment, AI_VISION_* authenticates against the Computer Vision resource used for generating embeddings, AI_SEARCH_* and AZURE_SEARCH_ADMIN_KEY connect to your search index, and ACCOUNT_KEY is used to generate SAS tokens for the storage account holding the images.
This is the part that does not line up with the architecture diagram from earlier. That diagram was built around managed identities and private endpoints, but the actual configuration here is entirely API key based, with keys sitting in plain environment variables. For a proof of concept that is a reasonable shortcut. For production, swap this for managed identity where the service supports it (Azure AI Search, Storage, and Azure OpenAI all support RBAC based access), and keep secrets that cannot avoid being keys in Key Vault with references from the Function App settings, not typed directly into App settings.
Deploying with GitHub Actions
The repository deploys the Function App automatically through a workflow file, main-premium.yml, that triggers on pushes to the main branch and uses the Azure Functions GitHub Action to publish the app. Two repository secrets are required for this to work: AZURE_RBAC_CREDENTIALS, a service principal with access to the subscription, and AZURE_FUNCTIONAPP_PUBLISH_PROFILE_PREMIUM, the publish profile for the Function App.
If you are setting this up fresh, I would skip the long lived service principal secret and use a federated (OIDC) credential instead. Azure AD app registrations support federated credentials tied to a specific GitHub repository and branch, which means there is no client secret sitting in your GitHub secrets store waiting to be leaked or to expire silently. It takes a few more minutes to configure once, and you stop having to rotate a credential every few months.
Inside the Function: the vectorize method
The vectorize method is what the AI Search skillset calls during indexing. It takes a batch of image URLs, generates a vector embedding for each one, and returns them as JSON so the indexer can write them into the search index. A caller (or the skillset, in production) hits it roughly like this.
# Example usage
image_urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]
embeddings = vectorize_images(image_urls)
print(embeddings)Under the hood, the POST body is parsed for the image URLs and metadata, and vectorize_images loops over each URL, calling a helper that first generates a SAS token for secure, time limited access to the image in Blob Storage, then calls get_image_embeddings against the Azure AI Vision endpoint to turn the image into a numeric vector that captures its visual features. The results are assembled into a single JSON response.
The detail that trips people up is that this endpoint does not write anything to the search index itself, it only returns embeddings. The actual write happens because the skillset treats this Function as a custom web API skill, so Azure AI Search calls it during indexing and handles persisting the output. If you test the endpoint directly with Postman and see a clean response but nothing shows up in your index afterward, check the skillset wiring and the indexer’s field mappings before assuming the Function is broken. Also watch your SAS token expiry if you are indexing a large batch of images. A short lived token that works fine for a handful of test images can start failing partway through a large indexing run if generation and processing take longer than the token’s validity window.
Inside the Function: the search method
The search method is the query side counterpart. It accepts a text query and an optional max_images parameter, and returns the closest matching images by vector similarity.
# Example usage
query = "Find images of mountains"
search_results = search_images(query, max_images=5)
print(search_results)The query first goes through ask_openai, which uses the Azure OpenAI chat deployment to rephrase the raw query into something that tends to search better. The refined query is then passed to generate_embeddings_text, which calls Azure AI Vision to produce a text embedding in the same vector space as the image embeddings, since Azure AI Vision supports multimodal embeddings where text and images can be compared directly. That embedding is wrapped in a VectorizedQuery object and handed to the Azure AI Search client, which returns the nearest image vectors along with freshly generated SAS tokens so the caller can actually load the images.
One trade off worth thinking about before you copy this pattern: refining every query through a chat completion call adds a full LLM round trip to your search latency, and it adds cost per search. That is fine for a low volume internal tool, but for a customer facing search box handling meaningful traffic, I would measure whether the refinement step meaningfully improves relevance for your image set before paying for it on every request. If it does not move the needle much, cut it and go straight from query text to embedding.
Testing the solution end to end
Testing has two halves that map to the two Function methods described above.
- Vectorize: upload images to the storage container, then run the indexer from the Azure AI Search portal so it pulls the images, calls the vectorize skill, and writes embeddings into the index. Watch the indexer’s execution history for failures.
- Search: send a query such as blue sky to the search endpoint through Postman or code, and confirm the returned results are visually related images with reasonable similarity scores.
What is deployed, and where this pattern fits
The sample resource group gives a useful sense of the real footprint of this solution: an AI Search service, an Azure OpenAI resource, a Computer Vision resource, a Premium plan Function App with its own App Service plan, a storage account for images plus a second one for the Function’s own runtime storage, a managed identity, an Event Grid topic, and Application Insights for monitoring.
This pattern fits well for catalog search on an e-commerce site, digital asset management for a media library, or an internal tool for finding similar product photos or design assets. It is a reasonable amount of infrastructure for a moderate volume, well scoped image collection, and the custom Function gives you full control over how embeddings are generated and how queries get refined.
It is worth less for very small image sets, where a simpler tag based or full text search would do the job with far less to operate and no per-request Azure OpenAI cost. It is also worth checking, since this article was published, whether Azure AI Search’s built in vectorizer support now covers your case directly in the skillset without a custom Function in front of it. Azure AI Search has been steadily expanding native vectorizer options for Azure AI Vision and Azure OpenAI embeddings, and if your requirements are simple text-to-image or image-to-image search without the query refinement step, you may be able to drop the custom Function entirely and cut one more moving part out of the architecture.
Closing thoughts
The value in this sample is less about the individual Azure services, which are all well documented on their own, and more about seeing how vectorize and search fit together as two ends of the same pipeline, one feeding the index and one querying it. If you are building something similar, start with the architecture diagram’s security posture rather than the sample’s default configuration, since private endpoints and managed identity are meaningfully more work to retrofit later than to design in from the start.
Leave a Reply