At Ignite 2024, Microsoft brought together a lot of loose ends in its AI tooling. Azure AI Studio gets renamed to Azure AI Foundry, and alongside the renamed portal, Microsoft ships the Azure AI Foundry SDK, a single coding surface that bundles Azure OpenAI, model inferencing, Azure AI Search, the Azure AI Agent Service, evaluation, and tracing.
If you have built AI applications on Azure over the last year, you already know the problem this SDK is trying to solve. You end up juggling the openai package, a separate Azure AI Search client, your own tracing wrapper, and possibly Semantic Kernel on top of all of it. The Foundry SDK tries to put most of this behind one project client, so switching between models or adding a new capability does not mean pulling in a new package and rewriting your configuration.
What actually ships in the SDK
The Azure AI Foundry SDK is available today in Python and C#, with a JavaScript version promised for later. If your backend runs on Node, that is worth flagging early, because you will either wait for the JS release or keep talking to the REST endpoints directly for now.
In this first release, the SDK covers the following pieces, all accessible from a single project client:
- Azure OpenAI Service, for GPT-4o, GPT-4o mini, GPT-4, DALL-E 3, Whisper, and the embeddings model family
- Model inferencing, a consistent client for models from OpenAI, Microsoft, Meta, Mistral AI, Cohere, and AI21 Labs
- Azure AI Search, for retrieval-augmented generation and hybrid search
- Azure AI Agent Service, for building tool-using agents without hand-rolling the orchestration loop
- Evaluation, with built-in and custom evaluators plus synthetic data generation
- Tracing, with OpenTelemetry instrumentation for the Inference, OpenAI, and LangChain SDKs
Trying models before writing any project code
The fastest way in is to just play with a model, and GitHub Models lets any developer do that for free without provisioning anything in Azure. If you work inside Visual Studio, installing the AI Toolkit for VS Code gets you the same model playground without leaving your editor.
This part is genuinely handy for a quick spike or a demo, since you can test a prompt idea in a couple of minutes and only move to an actual Azure AI project once you know the model behaves the way you expect.
Setting up an AI project
Azure AI projects consolidate everything you need to build an AI application in one place, and you start by creating a project in the Azure AI Foundry portal. From there, you install the projects package and connect using the connection string from the portal.
pip install azure-ai-projects azure-identity
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
project_connection_string = "your-connection-string"
project = AIProjectClient.from_connection_string(
conn_str=project_connection_string,
credential=DefaultAzureCredential()
)This block does one thing: it gives you a single project object that every other capability, OpenAI, inferencing, agents, evaluation, hangs off. The expected output is nothing printed to console, just a client ready to use. Two things trip people up here. First, do not hardcode the connection string in source, pull it from an environment variable or a secret store instead. Second, DefaultAzureCredential needs something to authenticate against, so run az login locally during development or rely on a managed identity once this is deployed to Azure.
Calling Azure OpenAI directly through the project
If you already have code written against the OpenAI SDK, you do not need to throw it away. Install the openai package and create an Azure OpenAI client from the project, and the rest of your calling code stays the same.
pip install openai
The get_azure_openai_client call wraps the standard openai package with your project’s credentials, endpoint, and API version, so you stop managing that configuration separately for every service you call. The response comes back in the familiar OpenAI chat completions shape. If you are migrating existing OpenAI SDK code, this is usually a two-line change: swap the client construction and keep your prompt and message-handling logic untouched.
Switching between models with the inference client
The model inference package gives you one interface for every model in the Azure AI model catalog, so you can compare providers without rewriting your calling code. Install the package first.
pip install azure-ai-inference
The client exposes a chat.complete() method that behaves the same way regardless of which model sits behind it. Watch what happens when you switch the model string.
Only the model name changes between the two snippets above, everything else, the client, the message shape, the response handling, stays identical. That is useful for benchmarking cost and quality across providers without maintaining separate integration code for each one. The catch is that this consistency is at the transport level. Provider-specific behaviour, such as how a model handles function-calling schemas or long system prompts, can still differ, so test each candidate model on your actual prompts before assuming full parity.
Prompt templates are the other useful piece here, letting you build a multi-turn prompt from named variables instead of string-concatenating everything by hand.
PromptTemplate.from_string parses a mustache-style template, and create_messages fills in the placeholders and returns the message list ready for a chat call. This keeps prompts readable and testable as templates instead of scattered f-strings, but remember it is still just string substitution under the hood. There is no automatic escaping, so validate or sanitise any user-supplied values before they land inside a template, especially if that value could contain instruction-like text.
Building agents without hand-rolling the loop
Plenty of teams already manually chain prompts, tool outputs, and function calls to build an agent that automates some business process. The Azure AI Agent Service is Microsoft’s attempt to make that pattern reusable, through a project client rather than a bespoke state machine.
This snippet uploads a document, creates a vector store from it, and attaches a FileSearchTool to an agent backed by gpt-4o-mini. The expected result is an agent object you can now run against user messages. A common mistake is treating upload_file_and_poll and create_vector_store_and_poll as fire-and-forget, they are polling calls precisely because indexing takes time, so do not assume the vector store is queryable the instant the call returns without checking status.
This follows the same thread and run pattern familiar from the OpenAI Assistants API: create a thread, add a message, kick off a run tied to your agent, then read messages back off the thread. The explicit check for run.status == “failed” matters more than it looks, skip it and a failed run silently returns whatever was in the thread before, which is a confusing bug to chase down in production.
On the enterprise side, the Agent Service lets you bring your own compute and storage, connect knowledge sources like Bing, SharePoint, and Microsoft Fabric, reach over 1,400 Logic Apps connectors, and deploy the resulting agent to more than fifteen Microsoft channels through the Microsoft 365 Agents SDK. If your team has already invested time in Semantic Kernel’s planner and agent abstractions, this is not a drop-in replacement, you are trading direct control over the orchestration loop for a managed service, and that trade is worth evaluating case by case rather than assuming the newer option is automatically better.
Instrumenting for observability
Tracing gives you visibility into what your app is actually doing across model calls, tool calls, and retrieval steps, which becomes essential the moment something behaves oddly in production. Enabling it is a couple of lines.
from azure.monitor.opentelemetry import configure_azure_monitor
# Enable OpenTelemetry instrumentation of the Inference, OpenAI and LangChain SDKs
project.telemetry.enable(destination=None)
# Log telemetry to the project's application insights resource
application_insights_connection_string = project.telemetry.get_connection_string()
if application_insights_connection_string:
configure_azure_monitor(connection_string=application_insights_connection_string)Calling telemetry.enable() instruments the Inference, OpenAI, and LangChain SDKs automatically, so you get spans without adding manual logging calls around every model invocation. On its own, destination=None only gets you local output, so pulling the Application Insights connection string and calling configure_azure_monitor is what actually persists traces centrally where your team can query them later.
That trace view is where the instrumentation earns its keep. You can see exactly which step in a retrieval-augmented pipeline added latency, whether it was the embeddings call, the search request, or the final generation step, instead of guessing from an aggregate response time. Do budget for Application Insights ingestion cost if you are tracing a high-volume production workload, since every span is billed telemetry.
Evaluating quality before and after shipping
The Azure AI Evaluation SDK is arguably the most immediately useful piece for teams already running a GPT-based app in production without any quality gates. Install it with the remote extra to get cloud-based batch runs.
pip install azure-ai-evaluation[remote]
This scores one query, response, and context triple against a built-in evaluator, here relevance, and returns a dictionary like {‘relevance.gpt_relevance’: 5.0}. Under the hood, RelevanceEvaluator itself calls a model to judge the answer, so factor that extra token cost in if you plan to run this on every request rather than a sample.
The batch version points evaluate() at a dataset path, maps evaluator inputs to your data columns through evaluator_config, and logs results against your Azure AI project so results are comparable run over run. Built-in evaluators cover groundedness, relevance, hate, and unfairness alongside industry metrics like ROUGE, BLEU, and F1, and the SDK also supports custom evaluators, synthetic data generation, user simulators, and adversarial simulations for jailbreak and safety testing.
Having these scores tracked centrally in the portal, rather than scattered across notebooks, is what makes evaluation part of a repeatable process instead of a one-time check before a demo. If your team has never run a formal evaluation pass on an LLM feature before, starting with groundedness and relevance on a small hand-picked dataset is a reasonable first step before investing in adversarial simulations.
Templates for deployment and GenAIOps
Deploying an AI app still means provisioning cloud infrastructure and wiring up a deployment pipeline, and the AI App Templates use infrastructure-as-code to shortcut that setup. The basic template deploys your code to a web app connected to your project resources with a couple of azd commands.
Beyond this basic template, more complete samples such as contoso-chat and contoso-creative-writer come with CI/CD, monitoring, and GenAIOps wired in already, which is a better starting point than the basic template if you are heading toward production rather than a proof of concept. There is also an Azure AI Evaluation GitHub Action that runs your evaluators on every commit, catching a quality regression before it reaches users instead of after. Automatic A/B experimentation and flighting on top of these evaluation checks is still in private preview at the time of this announcement, so treat it as a roadmap item rather than something to plan around today.
Who is already using it
Microsoft names a few early adopters alongside the announcement. Dentsu built a predictive analytics copilot on the SDK and Azure OpenAI Service that its team says cut time to media insight by ninety percent. C.H. Robinson used GitHub, Azure AI Foundry, and Azure OpenAI to automate email processing in logistics operations. Ontada applied the same stack to extract structured data from roughly 150 million unstructured oncology documents across thirty-nine cancer types. These are Microsoft’s own case studies rather than independent benchmarks, so read the specific percentages as directional rather than something you should expect to reproduce out of the box.
Should your team move to this now
If you are starting a new AI project in Python or C# with no existing investment elsewhere, the Foundry SDK removes a fair amount of plumbing, one client, one credential flow, and consistent patterns across models, search, agents, and evaluation. That is a real time saver compared to wiring these pieces together yourself.
If your team already has a working Semantic Kernel setup, there is no urgency to rip it out. Semantic Kernel still makes sense where you need fine-grained control over orchestration, and the Agent Service is a better fit for simpler tool-using agents where a managed run loop is enough. Since this SDK launched alongside a portal rename and several pieces, like automatic A/B experimentation, are still in private preview, pin your package versions and expect some API surface to shift between now and general availability. The missing JavaScript SDK is also worth flagging if your stack is Node-based, since you will be working against REST endpoints directly until that ships.
Leave a Reply