Most real business workflows do not fit inside a single prompt. A support ticket needs classification, a knowledge base lookup, a policy check, and sometimes a human sign off before anything gets closed. An invoice needs extraction, fraud screening, a limits check, and an approval gate before it touches the ERP. When I try to force this kind of process into one large agent with one giant system prompt, it gets brittle fast: the prompt grows unmanageable, failures are hard to trace back to a cause, and every small change risks breaking something unrelated. This is exactly the gap that multi-agent orchestration on Azure AI Foundry is built to close, and this article walks through how the pieces fit together and where I have seen teams get it right.
Why split work across agents at all
A single agent that tries to retrieve data, reason about it, check policy, and then act on a system of record ends up doing four different jobs badly instead of one job well. Splitting these responsibilities into separate agents gives you a natural fault boundary: if the retrieval agent starts returning stale results, you know exactly where to look, instead of debugging one enormous prompt. It also lets you pick the right model for each job. A cheap, fast model is often good enough for formatting or retrieval, while you reserve a stronger reasoning model for the step that actually needs it, which keeps your token bill sane.
Azure AI Foundry gives you three building blocks to do this without writing your own orchestration layer from scratch: the Agent Service for defining goal directed agents with tools attached, connected agents that let one agent call another as a tool using natural language (this is the A2A pattern, agent to agent), and multi-agent workflows that add a stateful layer on top for retries, compensation, and long running steps. On top of that, Foundry supports MCP (Model Context Protocol) for sharing context across agents and tools, plus built in tracing, evaluations, and content safety checks. None of this is unique to Foundry conceptually, but having it available as a managed service saves you from building and maintaining glue code yourself.
A reference architecture that actually holds up
The pattern I keep coming back to, and the one Microsoft’s own guidance leans on too, is an orchestrator agent that sits in front of specialist agents. A user or system event triggers the orchestrator, which delegates to a retrieval agent (pulling from Azure AI Search, Fabric, or OneLake), an analysis agent (reasoning or running code for calculations), a policy agent (checking entitlements, compliance rules, and approval thresholds), and an action agent (calling out to systems of record through OpenAPI, Logic Apps, or Azure Functions). The orchestrator consolidates the results, routes to a human approval step if the policy agent flags something, and every step gets logged for audit and telemetry.
The important design rule here, and one that is easy to skip when you are prototyping, is that each agent should own exactly one kind of responsibility. Retrieval agents should never be allowed to mutate systems, and action agents should not be making policy judgment calls on their own. Once you blur these lines, you lose the debuggability that was the whole reason to split the work in the first place.
Three scenarios worth building first
Customer support autopilot is the easiest entry point. The orchestrator receives a ticket, classifies priority and intent, the retrieval agent pulls the knowledge base and case history, the analysis agent drafts a fix, the policy agent checks entitlements and SLA terms, and the action agent updates the CRM and sends the response. The value here is mostly in root cause speed: when a support answer goes wrong, you can tell immediately whether it was a retrieval miss, a bad draft, or a policy check that should have blocked it.
Financial approvals is a heavier scenario and a good test of whether your workflow layer is doing its job. The orchestrator parses an invoice, an extraction agent pulls vendor and PO data, a risk agent screens for anomalies, a policy agent checks spend limits and compliance, and only after a human approval gate does the action agent post to the ERP. This is where the stateful workflow layer earns its keep, because you need retries that do not double post an invoice and a gate that genuinely blocks the action agent until a person has signed off.
Supply chain exceptions follow the same shape but reward parallelism more than the other two. The orchestrator ingests an exception, a forecast agent estimates impact, a vendor agent queries lead times over OpenAPI, a plan agent proposes a mitigation, a policy agent validates it, and the action agent places the change order. Running the forecast and vendor lookups in parallel rather than sequentially is what keeps latency reasonable, since the orchestrator does not need one result before starting the other.
Building your first connected agent workflow
You can build all of this either through the Foundry portal directly or through the Foundry SDK, and the steps mirror each other closely. Start by provisioning a project and your model deployments: a stronger reasoning model for the orchestrator, and a cheaper model for the specialists that do narrower jobs like retrieval or formatting. Connect your data sources, an Azure AI Search index or a Fabric or SharePoint source, and register the tools your action agents will need, such as OpenAPI specs, Logic Apps, or Functions.
Once the project is set up, define each agent’s role clearly before you write any code. The orchestrator understands goals, delegates, and composes the final answer. The retrieval agent grounds responses in enterprise content and nothing else. The analysis agent handles calculations or runs code for tabular work. The policy agent checks entitlements, data loss rules, and approval thresholds. The action agent is the only one allowed to call systems of record, and its calls should be auditable.
With roles defined, you register the specialists as connected agents on the orchestrator, which is the A2A part of the picture. In the portal this is just adding each specialist as a tool on the orchestrator’s configuration screen. In code it looks roughly like the snippet below. This is illustrative, not exact SDK syntax, since Foundry’s SDK surface changes across previews, but it shows the shape of what you are doing: creating each agent with its own tools, then connecting the specialists as callable children of the orchestrator.
# Conceptual snippet - refer to current Foundry SDK docs for exact classes
orchestrator = agents.create(
name="orchestrator",
instructions="Coordinate specialists. Delegate, verify, compile final answers."
)
retrieval = agents.create(name="retrieval", tools=["azure_ai_search:kb_index"])
analysis = agents.create(name="analysis", tools=["code_interpreter"])
policy = agents.create(name="policy", tools=["policy_rules:mcp"])
action = agents.create(name="action", tools=["openapi:erp", "logicapp:notify"])
# Connect specialists as tools on the orchestrator (this is the A2A wiring)
agents.connect(
parent=orchestrator.id,
children=[retrieval.id, analysis.id, policy.id, action.id]
)What this buys you is delegation by natural language rather than hand rolled routing logic. The orchestrator decides which specialist to call based on its instructions and the conversation so far, instead of you writing if-else branches for every possible request type. The trade-off is that you give up some predictability, so it is worth testing edge cases where the orchestrator might delegate to the wrong specialist, particularly early on before you have tuned its instructions.
Connected agents alone get you delegation, but production workflows also need state, retries, and a place to pause for a human. That is what the workflow layer adds on top. You define retry and backoff policies per step, run independent steps in parallel where it makes sense, and insert a human approval gate wherever the policy agent flags a condition, such as spend over a threshold. Again, this is conceptual, since the exact workflow definition format depends on which Foundry preview you are on.
# Pseudocode - actual workflow definition syntax depends on Foundry version
steps:
- delegate: retrieval
retry: { max: 2, backoff: exponential }
- parallel:
- analysis
- policy
- gate:
type: human_approval
condition: "policy.limit_exceeded == true"
- delegate: action
audit:
trace: enabled
pii_redaction: strictNotice the gate step sits between the parallel policy check and the action delegate, not before it. That ordering matters: you want the policy agent’s verdict available before deciding whether a human needs to look at it, rather than pausing for approval on every request regardless of risk. The retry policy on the retrieval step is also deliberate; retrieval failures are usually transient (a search index hiccup, a timeout), so a couple of retries with backoff clears most of them without bothering a human.
Once the workflow is running, use Foundry’s tracing, evaluations, and scoring to compare prompts, tool choices, and model combinations across agents. This is also where you attach cost and latency budgets per step, so you notice quickly if the analysis agent starts calling an expensive model for a task a cheaper one would handle fine, and where you turn on content safety filters before anything reaches production traffic.
Practical lessons from the field
Start with two agents, an orchestrator and one specialist, and prove the pattern works before adding more. It is tempting to design the full five-agent architecture up front, but most of that complexity only pays off once you have a concrete latency or clarity problem that a single specialist cannot solve. Adding agents before you need them just adds more places for things to go wrong.
Make every action idempotent, since retries will eventually happen whether you plan for them or not. A correlation ID passed through to the action agent is the simplest way to make sure a retried invoice posting does not create a duplicate entry in the ERP. Skipping this is one of the more common mistakes I have seen, usually discovered the hard way when a transient network blip causes a double charge.
Treat irreversible actions as a separate category that always needs a gate, log the full payload of every action call for audit purposes, and keep humans in the loop for approvals, exceptions, and SLA breaches with clear evidence attached rather than a bare notification. None of this is exciting work, but it is the difference between a demo that impresses people in a meeting and a system that survives contact with real production traffic and real compliance review.
When multi-agent is not worth it
Not every workflow needs this. If your task is genuinely a single lookup and response, for example answering a factual question from one knowledge base with no approval step and no side effects, a single agent is simpler to build, cheaper to run, and easier to reason about. Multi-agent orchestration earns its complexity when you have distinct responsibilities that benefit from isolation, real state that needs to survive retries, or a compliance requirement for human sign off. If you cannot point to which of those three applies to your workflow, it is worth questioning whether you need more than one agent at all.
Leave a Reply