Most Azure OpenAI proof of concepts follow the same path. A developer spins up an OpenAI resource, wires it to a chat UI, points it at a handful of documents in Azure AI Search, and the demo works well enough to impress a room. The trouble starts when that demo needs to become a production application inside an organization that already has a landing zone in place, with its own subscriptions, firewalls, DNS, and policy guardrails. Microsoft’s Azure Architecture Center addressed this gap with a reference architecture called the Azure OpenAI chat baseline architecture in an Azure landing zone, and it is worth understanding in detail if you are the one responsible for taking a generative AI pilot into a regulated enterprise environment.
Why a landing zone changes the picture
An Azure landing zone splits an organization’s cloud footprint into two categories of subscriptions. A platform landing zone holds centrally managed resources such as the hub network, Azure Firewall, DNS resolution, and connectivity to on-premises systems. An application landing zone is where a workload team deploys its own resources, and it connects back to the platform landing zone for networking, identity, and policy enforcement. If your organization already runs Azure landing zones for other workloads, an Azure OpenAI chat application does not get a free pass. It has to fit into the same governance model as everything else.
This matters because most public Azure OpenAI samples assume you own the entire subscription. You create your own virtual network, your own firewall rules, your own DNS zones. In a landing zone, a good chunk of that is owned by a platform team you do not report to, and changes to firewall rules or DNS routing can take days to land through a change process. Architects who skip this reality end up redesigning their network layer midway through a rollout, usually after a security review flags direct internet access from a resource that should have gone through a private endpoint.
What the baseline architecture actually deploys
Strip away the landing zone wrapper and the core application is fairly conventional. Azure Application Gateway with an integrated web application firewall sits at the front and terminates TLS. It routes traffic to an App Service instance running the chat UI, deployed across multiple availability zones. The retrieval augmented generation pattern is handled by Azure AI Search, which indexes your knowledge base and returns grounding data that gets stitched into the prompt before it reaches the Azure OpenAI model. Azure Key Vault holds the Application Gateway’s certificate, and Azure Monitor with Application Insights covers logging and diagnostics across every component.
Every one of these services, other than Application Gateway and the App Service front door, sits behind a private endpoint. Azure OpenAI, AI Search, Key Vault, and your storage account are not reachable from the public internet at all. This is the detail that turns a demo architecture into something a security team will actually sign off on, and it is also the detail that generates the most support tickets during initial rollout, usually because a DNS record for a private endpoint has not propagated yet.
Who owns what: the workload and platform split
The architecture is explicit about drawing a line between resources the workload team controls and resources the platform team controls. The workload team typically owns Azure OpenAI, AI Search, App Service, Key Vault, the application’s subnets, and workload-specific policies. The platform team owns Azure Firewall, Azure Bastion, the hub virtual network, private DNS zones for shared services, and any Azure Policy assignments applied at the management group level.
In practice this means a workload architect cannot simply add a new outbound dependency and expect it to work. If your chat application starts calling a third-party API for search grounding, that traffic has to route through the platform team’s egress firewall, and someone on that team has to add the corresponding allow rule. I have seen rollouts stall for a week over exactly this kind of dependency because nobody flagged it during the initial subscription vending conversation. The lesson is straightforward: list every outbound dependency your workload needs before you ask the platform team for a subscription, not after.
Networking: hub-spoke, DNS, and egress control
The application landing zone gets a spoke virtual network that peers with the platform team’s hub. All outbound traffic that leaves the spoke, including calls from App Service to Azure OpenAI over its private endpoint, follows user-defined routes that force it through the hub’s Azure Firewall. This is a deliberate design choice. It gives the platform team a single, centrally logged chokepoint for every packet leaving any workload in the organization, which is exactly what a security operations center wants when they are trying to correlate an incident across a hundred different application teams.
DNS resolution for private endpoints is where most of the friction actually shows up. In a self-contained subscription, you create your own private DNS zones and link them to your own virtual network, and resolution just works. In a landing zone, the platform team usually owns those DNS zones centrally, and your workload has to wait for the platform’s automation, often built on Azure Policy DeployIfNotExists rules, to create the A records for your private endpoints. If you deploy Azure OpenAI or AI Search before those DNS records exist, name resolution fails silently and the application throws confusing connection errors that have nothing to do with your actual code.
Policy conflicts you should expect, not be surprised by
Azure Policy assignments at the management group level do not know or care that your workload is a generative AI application. A common one is a policy requiring customer-managed keys for data at rest, which AI Search and Cosmos DB support but which the baseline architecture does not configure by default. Another is a policy blocking preview model deployments, which becomes a real problem during active development when the model you want to evaluate has not reached general availability yet.
None of these conflicts are architectural flaws. They are the normal cost of operating inside an organization that enforces consistent governance across every workload, AI or otherwise. The practical takeaway is to pull the list of policies assigned to your target management group before you start building, rather than discovering them when a deployment gets blocked by a policy you never knew existed.
Cost and operational trade-offs
Running inside a landing zone is usually cheaper at the margin, not more expensive, because Azure Firewall, DDoS Protection, and Azure Bastion get shared across every workload in the platform rather than provisioned per application. The published cost estimate for the baseline architecture does not include this shared platform infrastructure, so do not compare it directly against a standalone deployment without accounting for the chargeback or shared-cost model your organization uses.
Where the landing zone pattern costs you is in lead time and flexibility. Every new outbound dependency, every DNS zone, every firewall rule change becomes a conversation with another team instead of a configuration change you make yourself. For a small internal tool with a handful of users, that overhead is rarely worth it, and a simpler standalone deployment with its own network isolation will get you to production faster. For a customer-facing chat application handling regulated data across a large organization, the landing zone model is the difference between passing a security review on the first attempt and failing it repeatedly.
Getting started without redesigning everything
Microsoft publishes a companion GitHub sample, the Azure-Samples openai-end-to-end baseline repository, that deploys this architecture with infrastructure as code. It is a reasonable starting point for a proof of value exercise, but treat it as a reference rather than something you deploy unmodified into a shared subscription. Your platform team’s naming conventions, subnet sizing, and policy exceptions will not match the sample’s defaults, and reconciling those differences is exactly the negotiation the architecture expects you to have before deployment, not after.
If your organization has not yet adopted Azure landing zones at all, this architecture is still useful reading even though most of the platform-team sections will not apply to you directly. It lays out a sensible default for network isolation, RAG grounding, and monitoring that holds up regardless of whether a separate platform team enforces it or you enforce it yourself within a single subscription.
Leave a Reply