Vision models are usually shown doing one thing: describing what is in a picture. A more useful test is whether they can turn a photo into data your application can actually use. Grocery receipts are a good candidate for this, since a receipt photo has clear structure, item names, quantities, prices, but that structure is buried inside a JPEG rather than sitting in a database column.
The goal here is straightforward. Take a photo of a grocery receipt, run it through a vision model, and get back line items, quantities and prices as strongly typed C# objects. Everything runs locally with Ollama and a llama3.2-vision model, so there is no API key to manage and no image data leaving the machine.
Setting Up Ollama With Microsoft.Extensions.AI
Ollama lets you run large language models on your own hardware. You pull a model the way you would pull a Docker image, then run it as a local service.
ollama pull llama3.2-vision:latest
# Then run the model locally
ollama run llama3.2-vision:latestThe pull downloads the model weights once. After that, ollama run keeps the model loaded and answers requests over its local API on port 11434. On a laptop with a decent GPU this works fine. On CPU-only hardware, expect responses to take noticeably longer, since vision models are larger than their text-only counterparts.
On the .NET side, Microsoft.Extensions.AI gives you a single IChatClient interface that works the same way regardless of which model provider sits behind it. Paired with the OllamaSharp package, wiring this up takes only a few lines.
var builder = Host.CreateApplicationBuilder();
builder.Services.AddChatClient(
new OllamaApiClient(
new Uri("http://localhost:11434"),
"llama3.2-vision:latest"));
var app = builder.Build();
var chatClient = app.Services.GetRequiredService<IChatClient>();This registers an OllamaApiClient pointed at the local Ollama endpoint and the llama3.2-vision model, then resolves it as an IChatClient from the DI container. Nothing here talks to a cloud API, so there are no secrets or connection strings to configure for this part.
The real benefit of IChatClient is that it is provider-agnostic. If you later decide to swap Ollama for Azure OpenAI or GitHub Models, the rest of your application code stays exactly as it is. Only this registration changes.
Sending an Image to the Model
The simplest thing to try first is asking the model to describe what is in a photo, without asking for any specific format.
var message = new ChatMessage(
ChatRole.User, "What's in this image?");
message.Contents.Add(
new DataContent(
File.ReadAllBytes("receipts/receipt_1.png"),
"image/png"));
var response = await chatClient.GetResponseAsync([message]);
Console.WriteLine(response.Text);You read the image bytes from disk, wrap them in a DataContent object along with the correct MIME type, and attach that to a ChatMessage. The chat client sends the text prompt and the image bytes to the model in a single request.
Here is the raw text the model returned for this receipt:
This image appears to be a receipt or invoice in a foreign language, likely
Russian or another Slavic language. The document is in black and white and
features a large QR code at the bottom. The text is in a blocky, old-style
font and includes several columns of numbers and words. The top of the page
has a header with some information in a foreign language, followed by a
series of columns with various details such as date, time, and product
information. The document also includes some calculations and a total at
the bottom.The model correctly recognised it as a receipt and picked up on the general language and layout, which is a reasonable result for a first attempt with no fine-tuning. But a paragraph of prose is not something you can feed into an expense-tracking database. For that, the model needs to return a specific, parseable format.
Asking for JSON Output
The fix is to describe the exact JSON shape you want directly in the prompt.
var message = new ChatMessage(ChatRole.User,
"""
Extract all line items from this receipt.
Respond in JSON format with this structure:
{
"items": [
{
"name": "item name",
"quantity": 1.500,
"unitPrice": 0.00,
"totalPrice": 0.00
}
],
"subtotal": 0.00
}
""");
message.Contents.Add(
new DataContent(
File.ReadAllBytes("receipts/receipt_1.png"),
"image/png"));
var response = await chatClient.GetResponseAsync([message]);
Console.WriteLine(response.Text);The prompt gives the model both the instruction, extract line items, and the exact schema to follow, down to field names and example values. This matters more than it looks. Vision models tend to follow a schema closely if you show them one, but improvise field names if you only describe the shape in words.
Here is what came back for the same receipt:
{
"items": [
{
"name": "limun /kg (A)",
"quantity": 280.00,
"unitPrice": 1.105,
"totalPrice": 309.40
},
{
"name": "salata /kom (A)",
"quantity": 70.00,
"unitPrice": 3.00,
"totalPrice": 210.00
},
{
"name": "susam 100g trpeza /kom (A)",
"quantity": 90.00,
"unitPrice": 1.00,
"totalPrice": 90.00
}
],
"subtotal": 609.40
}This worked far better than expected on the first attempt. Item names, quantities and prices came back in a shape you could deserialize directly. A few quantities were slightly off, which is common when a vision model reads small printed text, but the overall structure held up.
Iterating on the System Prompt
Once basic JSON extraction works, most of the remaining effort goes into the system prompt rather than the C# code. When the model misreads a digit or invents an item, you do not fix that with a code change. You fix it by telling the model, in plain language, what mistake to avoid.
A system prompt is the initial instruction that sets context for the whole conversation, separate from the actual question being asked. In code, this is the message using ChatRole.System, while the receipt question itself uses ChatRole.User. After a few rounds of trial and error, the system prompt for this receipt scanner ended up reading closer to a specification document than a casual instruction:
var systemMessage = new ChatMessage(ChatRole.System,
"""
You are a receipt parsing assistant. Extract all line items from
the receipt image. For each line item, extract the name, quantity,
unit price, and total price. Quantity can be a decimal number
(e.g. weight in kg like 0.550 or 1.105). Extract the subtotal,
which is the final total amount shown on the receipt.
IMPORTANT: Read every digit exactly as printed on the receipt.
Pay very close attention to each decimal digit, do NOT round or
approximate. For example, if the receipt shows 1.105, report
exactly 1.105, not 1.1 or 1.2. Verify that quantity times
unitPrice equals totalPrice for each line item. Don't invent
items that aren't on the receipt.
DECIMAL FORMAT: Some receipts use a comma as the decimal
separator instead of a period. Treat 1,105 the same way you
would treat 1.105, and do not confuse it with a thousands
separator.
""");Every line in that prompt exists because of a specific failure in an earlier run. The instruction to read every digit exactly as printed was added after the model rounded 1.105 down to 1.1. The line about not inventing items was added after it hallucinated a line item that was not on the receipt at all. The section on decimal formatting exists because these particular receipts use a comma as the decimal separator, and the model kept confusing that comma with a thousands separator.
This is worth calling out as a general pattern when you work with vision or language models for extraction tasks. Prompt iteration is a debugging process, just conducted in English instead of C#. It takes longer than fixing a compiler error, and there is no stack trace to point you at the problem, only a wrong answer and your own judgement about why it went wrong.
Getting Strongly Typed Responses
Parsing raw JSON strings by hand works, but Microsoft.Extensions.AI has a cleaner option. Calling GetResponseAsync<T> instead of GetResponseAsync returns a strongly typed object directly, with the library generating a JSON schema from your type and handling deserialization itself.
var response = await chatClient.GetResponseAsync<Receipt>(
[systemMessage, message],
new ChatOptions { Temperature = 0 });
if (response.Result is { } receipt)
{
Console.WriteLine($"\nExtracted {receipt.Items.Count} line items:");
foreach (var item in receipt.Items)
{
Console.WriteLine(
$" {item.Name} - " +
$"Qty: {item.Quantity} x {item.UnitPrice:C}" +
$" = {item.TotalPrice:C}");
}
Console.WriteLine($" Subtotal: {receipt.Subtotal:C}");
}Setting Temperature to 0 pushes the model toward more deterministic output, which matters for an extraction task where you want the same input to produce the same answer. It is not a guarantee, since vision models are not perfectly deterministic even at temperature 0, but it reduces the variance noticeably.
The Receipt and LineItem types behind this call are ordinary C# classes, nothing special:
public class Receipt
{
public List<LineItem> Items { get; set; } = [];
public decimal Subtotal { get; set; }
}
public class LineItem
{
public string Name { get; set; } = string.Empty;
public decimal Quantity { get; set; }
public decimal UnitPrice { get; set; }
public decimal TotalPrice { get; set; }
}There is no attribute magic here and no custom converter to write. Microsoft.Extensions.AI reflects over these types, builds a schema the model can follow, and hands back a populated Receipt instance. Here is what that object looks like when inspected in the Visual Studio debugger:
Checking Whether the Output Is Consistent
Getting a correct answer once is encouraging, but an extraction pipeline needs to behave the same way every time it sees the same input. So the next check was to send the same receipt with the same prompt five times and compare the results.
const int runs = 5;
var results = new List<Receipt>();
for (int i = 0; i < runs; i++)
{
var testResponse = await chatClient.GetResponseAsync<Receipt>(
[systemMessage, message],
new ChatOptions { Temperature = 0 });
if (testResponse.Result is { } r)
{
results.Add(r);
Console.WriteLine(
$" Items: {r.Items.Count}, Subtotal: {r.Subtotal:C}");
}
}Each run stores its Receipt result in a list. With temperature set to 0, you would expect these five runs to agree completely, since the same prompt and the same image go in every time.
The comparison itself checks item counts, subtotal, and every individual line item field against the first run:
var baseline = results[0];
for (int i = 1; i < results.Count; i++)
{
bool match = baseline.Subtotal == results[i].Subtotal
&& baseline.Items.Count == results[i].Items.Count
&& baseline.Items.Zip(results[i].Items).All(pair =>
pair.First.Name == pair.Second.Name
&& pair.First.Quantity == pair.Second.Quantity
&& pair.First.UnitPrice == pair.Second.UnitPrice
&& pair.First.TotalPrice == pair.Second.TotalPrice);
Console.WriteLine(
$" Run 1 vs Run {i + 1}: {(match ? "MATCH" : "DIFFERENT")}");
}Most runs matched exactly. A few did not, usually differing by a misread digit or a slightly reworded item name. This is a useful reminder that even with temperature pinned to zero, vision models remain probabilistic systems rather than deterministic functions. If your use case needs guaranteed accuracy, such as feeding numbers into an accounting system, add a validation layer on top of this rather than trusting the model’s output directly, for instance checking that quantity times unit price equals total price for each line and flagging anything that does not.
Where This Can Go From Here
A working receipt scanner is a starting point, not an end product. Once receipt images turn into structured data, a natural next step is a personal finance tracker: scan receipts, store the parsed items, and let the same model categorise purchases into groceries, household or electronics.
From there, weekly and monthly spending summaries follow fairly directly, along with multi-receipt aggregation for something like a business trip, where an expense report gets generated from a stack of scanned receipts instead of one at a time. Price tracking over time, noticing when a specific item at your regular store creeps up in price, is another reasonable extension, as is semantic search over past receipts using embeddings and vector search, so you can ask which receipts included a particular item without scanning through them manually.
Before building any of this into something people rely on, it is worth being clear about the trade-offs of the local-first approach used here. Running llama3.2-vision locally means no per-request API cost and no image data leaving your machine, which matters if the receipts contain anything sensitive. The cost is accuracy and speed. A local vision model on consumer hardware will generally lag behind a hosted model such as GPT-4o vision on tricky handwriting or low-quality scans, and inference will be slower unless you have a capable GPU. For a personal project or an internal tool with a small number of users, local inference is a reasonable default. For a customer-facing product handling thousands of receipts a day, a hybrid setup, where local inference handles the common case and falls back to a cloud vision model for images the local model is unsure about, is worth evaluating instead.
Summary
Running a vision model locally with Ollama is not complicated to set up, and Microsoft.Extensions.AI together with OllamaSharp keeps the .NET side of it clean. You end up with a provider-agnostic IChatClient that supports strongly typed responses through GetResponseAsync<T>, without writing your own JSON parsing or schema code. If you would rather stay entirely within the Microsoft ecosystem, this same pattern works with Aspire and GitHub Models as well.
The system prompt is where most of the actual engineering effort goes. In this experiment, every line in the final prompt was a direct response to a specific mistake the model made in an earlier run, not something written up front.
If you want to try this yourself, here is a reasonable starting sequence:
- Install Ollama and pull the vision model with ollama pull llama3.2-vision:latest
- Create a .NET console app and add the OllamaSharp and Microsoft.Extensions.AI NuGet packages
- Wire up an IChatClient pointed at your local Ollama endpoint
- Point it at a receipt image and see what comes back, then refine your system prompt based on what goes wrong
Leave a Reply