Innovate with Azure Assistant API

Most chatbot projects stop at information retrieval. A customer asks about a policy or a product, the bot pulls an answer from a knowledge base, and the conversation ends there. But real customer requests rarely stop at ‘tell me’.

Consider a credit card holder who missed an EMI payment. He does not just want to know the outstanding amount and the penal interest applied to it, he wants the amount calculated in real time and the payment settled in the same conversation. That means pulling policy details, calling third party APIs for account information, computing the net due amount, processing the payment, and logging the interaction in the CRM.

This is where the Azure Assistants API becomes useful. It is built to combine reasoning, computation, and tool calling in one place, instead of making you stitch these pieces together yourself.

What the Assistants API actually gives you

The Assistants API bundles three capabilities that used to require separate plumbing: a code interpreter for real computation, custom functions you define yourself, and a retrieval mechanism for grounding answers in your own documents. All three sit inside a thread management layer that keeps conversation context without you having to resend the entire chat history on every call. You can attach up to 128 tools to a single assistant and they can run in parallel, and the assistant accepts a fairly wide range of file formats including CSV, TXT, PDF, DOC, and JSON. It can also return generated images and CSV files as output, not just text, which matters when the code interpreter produces a chart or a computed table.

This is a meaningful difference from a plain chat completion call. With chat completions, you own the orchestration: deciding when to call a function, feeding results back into the prompt, and managing how much history to send. The Assistants API moves that orchestration into the platform. You describe the tools, and the assistant decides when to invoke them during a run.

Setting up the tools

For the EMI scenario, the assistant needs two supporting files: one with financial product information, and one with the interest charged on late EMI payments across different products. These are uploaded to Azure OpenAI so the assistant can reference them during a run.

filePath = DATA_FOLDER + filename
with Path(filePath).open("rb") as f:
    new_file = client.files.create(file=f, purpose="assistants")
    assistant_files.append(new_file.id)

This loop reads each file in binary mode and uploads it with purpose set to assistants, which is what tells Azure OpenAI the file is meant to support an assistant run rather than a fine-tuning job. The returned file ID goes into assistant_files, a list you pass in later when the assistant is created. A common mistake here is forgetting that uploaded files count against storage and retrieval limits on your Azure OpenAI resource, so periodically checking what is uploaded and deleting files you no longer need is worth building into your operations, not treating as an afterthought.

Next comes a custom function that categorizes the incoming user query, so the assistant knows whether it is dealing with a generic question, a service request, or a personal information lookup.

available_functions = {"categorize_user_query": categorize_user_query}

This dictionary maps a function name string to the actual Python callable. The Assistants API works with function names as strings when it decides to call a tool, so you need this lookup table to resolve the name back to code you can actually execute. If you add more custom functions later, they all go into this same dictionary, and it is easy to forget to register one, which then shows up as a silent no-op rather than an obvious error.

The third tool is the built-in code interpreter, which does not need a Python function of your own. You just add it to the tools list when you create the assistant, and Azure OpenAI provisions a sandboxed environment where it can write and execute code, including tasks like the EMI penalty calculation in this example.

Creating the assistant

With the files uploaded and the function registered, the assistant itself is created with a name, instructions, the list of tools, the target model deployment, and the file IDs.

assistant = client.beta.assistants.create(
name=name,
instructions=instructions,
tools=tools,
model=open_ai_deployment_name,
file_ids = assistant_files
)

The instructions parameter deserves more attention than it usually gets. This is where you tell the assistant its role, the tone it should use, and any hard constraints, such as never disclosing another customer’s account details. Vague instructions lead to an assistant that wanders between tools unpredictably, so it is worth treating this like a short system prompt you iterate on rather than a one-line description you write once and forget.

Threads, messages, and runs

An assistant on its own does not hold a conversation. That is the job of a thread, which is created once per conversation and accumulates messages over its lifetime.

thread = client.beta.threads.create()

Threads are cheap to create but not free to keep around indefinitely. If your application spins up a new thread per session and never cleans up old ones, you will accumulate orphaned threads in your Azure OpenAI resource. Worth adding a cleanup job that deletes threads past a certain age, especially in a production deployment serving many concurrent users.

Once the thread exists, the user’s question, and optionally a file, is added to it as a message.

return client.beta.threads.messages.create(
    thread_id=thread_id, role=role, content=content)

The role parameter is almost always “user” for incoming queries, and content carries the actual text. You can also attach files at the message level, not just at assistant creation, which is handy when a user uploads a document mid-conversation rather than the assistant needing it from day one.

With the message in place, a run is created and polled until it finishes.

while status ==0:
    run = client.beta.threads.runs.create(
        thread_id=thread.id, assistant_id=assistant.id,
        instructions=instructions)

A run is where the actual work happens: the assistant reads the thread, decides which tools to call, executes them, and produces a response. Polling in a tight loop like this is fine for a demo, but in a production API you would want exponential backoff on the poll interval and a timeout, because a run stuck in a queued or in_progress state for too long usually points to a rate limit or a quota issue on the underlying model deployment rather than something wrong with your code.

Reading back the response

Once a run completes, the assistant’s reply sits in the thread as one or more messages, and it can contain both text and generated image files, since the code interpreter can produce charts or plots as part of its answer.

This function lists all messages on the thread and walks them in reverse, since the most recent assistant message is what you usually want to surface first. It skips the user’s own message, then for each remaining content block, it checks the type: text content gets base64 encoded and appended to final_response, and image_file content gets its bytes pulled through client.files.content and encoded the same way. The output is a list of dictionaries your front end can render directly, whether that is a chat bubble with text or an inline chart image.

One thing to watch for: the image_file reference is only a file ID, so you need a separate call to fetch the actual bytes. Skip that step and you will end up rendering a file ID string as if it were image data, which fails silently in some UI frameworks and loudly in others.

Where this fits and where it does not

The Assistants API is a good fit when your bot genuinely needs to reason across multiple tools in a single turn, particularly when computation or code execution is involved, like the EMI interest calculation here. It is a poor fit if your use case is a straightforward FAQ bot answering from a fixed knowledge base, where a simpler retrieval-augmented setup without thread and run management will be cheaper to run and easier to debug. The same pattern shown for BFSI here maps to HR bots answering vacation balance or leave request queries, and to retail bots handling return status or order tracking, since the underlying shape of the problem, classify the query, call the right tool, respond, is the same across verticals.

One trade-off worth flagging before you commit to this architecture in production: the polling model shown in the run creation step adds latency compared to a direct function call, and every run consumes tokens for the assistant’s internal reasoning steps, not just the final answer. For high-volume, low-complexity queries, that overhead adds up in both response time and cost, so it is worth measuring against your actual traffic pattern before assuming the Assistants API is the right tool for every request type your bot handles.