When Microsoft first released the Assistants API on Azure OpenAI Service in early 2023, it removed a fair amount of plumbing that developers earlier had to build by hand: thread management, tool orchestration, and state handling for conversational agents. At Microsoft Build 2024, the team pushed out a preview refresh of Assistants that adds two new tools, tightens up enterprise security controls, extends model support, and cleans up several rough edges that early adopters had been running into. This piece walks through what actually changed, why each piece matters for a production rollout, and where the gaps still are.
File Search: retrieval without hand-rolling a vector pipeline
The headline addition is File Search, now in public preview. Before this, if you wanted an Assistant to answer questions grounded in your own documents, you had to stand up your own chunking, embedding, and vector store pipeline, then wire it into the Assistant through function calling. File Search folds that entire pipeline into the API itself. You attach files to an Assistant, and Azure OpenAI handles parsing, chunking, embedding, and indexing behind the scenes.
A new object called vector_store sits at the center of this. Once a file lands in a vector store, it gets parsed, chunked at a default size of 800 tokens with a 400 token overlap, and embedded using text-embedding-3-large at 256 dimensions. Vector stores are reusable across multiple Assistants and threads, which matters operationally because you are not re-embedding the same document set for every Assistant you spin up. File Search supports up to 10,000 files per Assistant and runs multi-threaded queries in parallel, with reranking and query rewriting built in rather than left for you to implement.
The chunk size and overlap defaults are worth paying attention to before you commit to this for a production RAG workload. An 800 token chunk with 400 token overlap is a reasonable general-purpose setting, but it is not tunable at the time of this refresh, and that is a real constraint if your documents are dense technical specs or long-form legal text where a smaller or larger chunk changes retrieval quality noticeably. If you already have a custom chunking strategy tuned for your content, File Search will not let you bring it over yet.
Pricing is straightforward: $0.10 per GB of vector store storage per day, with the first GB free, and Microsoft ran File Search at no cost until June 17, 2024 as an introductory period. If you are prototyping with File Search now, budget for storage costs once that window closes, especially if you are storing large document sets across many vector stores rather than consolidating them.
A related feature called Bring Your Indexes to File Search was announced for public preview in July 2024. This lets teams who already built retrieval pipelines through Azure OpenAI On Your Data, against sources like Azure AI Search, Cosmos DB for MongoDB vCore, Blob Storage, Pinecone, or Elasticsearch, point File Search at those existing indexes instead of re-ingesting everything. That is a meaningful convenience if you already invested in an On Your Data setup and do not want to duplicate storage and embedding costs.
Browse: web search as a first-class tool
The second new tool is Browse, announced for public preview in July 2024. It lets an Assistant search the web through Bing when a user’s question needs current information the model was not trained on, things like today’s weather or a recent news event. The Assistant formulates a keyword query from the user’s question and submits it to Bing automatically.
Browse is useful for the narrow case of freshness, but it is not a general web browsing agent that can navigate multi-step tasks across pages. Treat it as a retrieval tool for current facts, not a replacement for a dedicated browsing automation solution if your use case needs to fill forms or follow links across a session.
Locking down enterprise data: CMK, token controls, and prompt injection defenses
For teams that need to clear a security review before Assistants goes anywhere near production, this refresh addresses three gaps that were previously blockers.
Customer-managed key support for Assistants thread state and files, in public preview from June 2024, lets you control encryption keys for stateful data through Azure Key Vault or a managed HSM instead of relying solely on Microsoft-managed keys. CMK support for File Search specifically was still coming at the time of this refresh, so if your compliance requirement covers retrieval data as well as thread state, confirm that gap is closed before you commit to a launch date.
Token and cost controls are the more immediately useful addition for most teams. You can now see input and output token usage at the thread, message, and run level, cap the maximum tokens a run can consume, and limit how many previous messages get pulled into context for a given run. There is also a new tool_choice parameter that lets you force a specific tool, say file_search, for a given run instead of leaving tool selection entirely to the model. This matters because Assistants threads accumulate history over a conversation, and without a cap on message count or token budget, a long-running thread can quietly become an expensive run. Setting these limits early, rather than after your first surprising bill, is the more sensible order of operations.
The security addition worth understanding in more depth is protection against cross-prompt injection attacks, or XPIA. This is a real and underappreciated risk category for any Assistant that summarizes documents, reads web pages, or describes images as part of its job. An attacker can embed instructions inside that external content, and the model may misinterpret those embedded instructions as commands from the developer or user and act on them. The refresh adds support for prompt shields, which specifically detect and block this kind of indirect injection.
You turn this on through the Content Filter Creation wizard in Azure AI Studio, on the model deployment your Assistant uses, by enabling the prompt shield for indirect attacks. It is not enabled by default, so if your Assistant processes any content it did not author itself, uploaded documents, scraped pages, image descriptions, this is a setting to check explicitly rather than assume is already active. Prompt shields also help against regurgitation of protected code or text, which is a separate but related concern if your Assistant has access to proprietary source or licensed content.
Wider model support: GPT-4o, vision, and fine-tuned models
The refresh brings gpt-4o support to Assistants, in public preview from June 2024. GPT-4o handles text and images in a single model rather than routing image understanding through a separate model, which simplifies building a multimodal Assistant considerably. Alongside that, vision support arrives through gpt-4-turbo (0409), letting you attach image URLs or uploaded files to a message so the Assistant can factor visual content into its response.
Fine-tuned gpt-35-turbo (0125) is now usable with Assistants, though only in Sweden Central and East US2 at this point. If your fine-tuned model lives in a different region, you cannot yet attach it to an Assistant, so this is worth checking before you plan a rollout around a fine-tuned model. Regional availability for Assistants generally also expanded to Japan East, UK South, West US, and West US3, which is useful if data residency requirements have been keeping you out of the earlier regions.
Developer experience: streaming, SDKs, tracing, and multi-agent patterns
A handful of changes target the gap between a working prototype and something you would actually ship. Streaming support means responses can now be streamed back to the client instead of waiting for a full run to complete, which noticeably reduces perceived latency in a chat interface. The Python SDK adds a create-and-stream helper for this, along with helpers that push object status updates instead of requiring you to poll the API in a loop, which was previously the only way to know when a run had finished.
SDK coverage also widens. Python and JavaScript or TypeScript support is available now through the OpenAI client libraries, .NET support was slated for June, and Java and Go for July. If you are building on .NET or Java today, this refresh is more of a signal that support is coming than something you can use immediately, so plan your timeline accordingly rather than assuming parity across languages.
For teams past the prototype stage, PromptFlow SDK integration adds tracing and evaluation for Assistants, in public preview from June 2024. Tracing instruments your tool and function calls so you can visualize the execution flow of a run, locally or in Azure AI Studio, which is genuinely useful once an Assistant is calling multiple tools per turn and you need to work out where a run is spending time or going wrong.
The evaluation side ships built-in evaluators for both quality (groundedness, relevance, coherence, fluency, similarity, F1 score) and risk and safety (violence, sexual content, self-harm, hate and unfairness). This is the kind of tooling that is easy to skip during a demo and expensive to skip once an Assistant is live and answering real user questions with no structured way to catch a regression after a prompt or model change. If you are running Assistants in any customer-facing capacity, wiring evaluation into your deployment pipeline from the start is worth the upfront effort.
Two integration patterns round out this section. Azure OpenAI Assistants can now participate in AutoGen multi-agent workflows through a new GPTAssistantAgent, letting specialized Assistants collaborate on a task rather than one Assistant trying to do everything. Separately, Azure Logic Apps workflows can be imported directly into the Assistants playground as callable functions, with the function specification pulled automatically from the workflow’s swagger definition. That second one is a genuinely practical shortcut if your organization already has Logic Apps automating business processes, since you skip writing function schemas by hand and get an existing, tested workflow as an Assistant tool.
How this is landing with early adopters
Microsoft highlighted a few customer deployments alongside the announcement. Coca-Cola built an internal tool called KO Assist on top of the Assistants API, using Code Interpreter and File Search to give roughly 30,000 associates access to business insights and enterprise data without a lengthy engineering build, reporting weeks rather than months to get it operational. Freshworks built its Freddy AI platform on Assistants, using File Search and parallel function calling so customers can stand up AI agents against their own data with close to no configuration.
Microsoft Copilot for Finance, which connects to systems like Dynamics 365 and SAP, uses the Assistants API and its Code Interpreter tool specifically for variance analysis, uploading and transforming financial data and running exploratory calculations inside the run itself. The common thread across all three is that Code Interpreter and File Search, rather than custom function calling, are doing the heavy lifting, which lines up with Microsoft’s stated goal of reducing the amount of custom orchestration code teams need to write.
What this means if you are evaluating Assistants today
This refresh moves the Assistants API meaningfully closer to production readiness, but it is still a preview product, and several of the features described here, Browse, CMK for File Search, .NET and Java SDK support, were still rolling out rather than generally available at the point of this announcement. Before you commit a production workload to Assistants, check the current status of the specific feature you depend on against the Azure OpenAI what’s new page, since preview timelines shift.
If your use case is retrieval-augmented generation against enterprise documents, File Search removes a genuine amount of infrastructure work, but the fixed chunking strategy is a real limitation if you have already tuned a custom pipeline. If your Assistant touches any content it did not author, enable the XPIA prompt shield rather than assuming default settings cover you. And if you are moving past a demo into something users will rely on, set token and message limits explicitly and wire in PromptFlow evaluation before you need it, not after a bad response surfaces in production.
Leave a Reply