At Microsoft BUILD 2023, the Semantic Kernel team followed their keynote, Building AI solutions with Semantic Kernel, with a live Q&A session. The questions from that session tell you more about real world LLM app development than most marketing content ever will. Below I have expanded on five of those questions with the practical detail a working developer needs.
Picking a vector store and chunking strategy for document Q&A
One attendee asked whether AI could fill in sections of new documents based on historical examples, a common requirement in insurance, legal and consulting work. The Semantic Kernel team’s answer breaks the problem into three decisions: pick a vector memory store, decide how you will chunk large documents before embedding them, and design the UI your end users will actually touch.
Semantic Kernel ships connectors for Azure Cognitive Search (C#), Chroma (Python), Pinecone (C#), Postgres (C#), Qdrant (C#), Redis (C#), SQLite (C#) and Weaviate (both C# and Python). That is a wide spread, and the right choice depends less on the framework and more on what you already operate. If your team already runs Postgres or Redis in production, use the existing connector rather than adding a new managed vector database purely because it is trendy.
Chunking strategy gets glossed over quickly in the answer, but it deserves more attention than it gets. Split documents too aggressively and you lose the surrounding context a chunk needs to make sense on its own; split them too coarsely and you blow past the token budget of your embedding model or drown the LLM in irrelevant text. A sensible starting point is chunking by section or heading rather than a fixed character count, then testing retrieval quality against real questions before you commit to a scheme.
The team points to the Copilot Chat sample app as a working reference. It lets a user upload a file, which then gets embedded and stored in whichever vector store is configured. That sample is a reasonable starting point to fork, though production teams will want to add their own error handling around upload failures and embedding rate limits.
Letting employees query enterprise SQL data without leaking rows
The second question is one every enterprise AI team eventually runs into: how do you let employees talk to data sitting in SQL without opening the door to prompt injection or data leakage across users? The answer given is refreshingly practical. Authenticate the user first, then pass that identity through to the database so access control happens at the data layer, not inside the prompt.
This matters because relying on the LLM itself to enforce row level security is fragile. A cleverly worded prompt can talk a model into ignoring instructions it was given in the system prompt, but it cannot talk its way past a database permission it was never granted. Using views and stored procedures instead of letting the LLM generate raw SQL statements closes off an entire class of injection attacks, since the model is choosing from a fixed set of safe operations rather than composing arbitrary queries.
In practice this means your Semantic Kernel plugin functions should call parameterized stored procedures scoped to the authenticated user’s tenant and role, not a generic ‘run this SQL’ function. It is more setup work up front, but it turns a security review from a lengthy back and forth into a much shorter conversation.
Keeping AI answers consistent with static plans
Consistency is the third theme, and it is a real pain point once you move past a demo. Ask the same question twice and an LLM can reasonably give you two different sequences of steps to answer it, which is unacceptable in a business process. The team’s suggestion is to create static plans using the Semantic Kernel VS Code extension, then reuse that fixed plan every time a matching request comes in instead of asking the planner to improvise a new one.
This is essentially trading a small amount of flexibility for a large amount of predictability. Dynamic planning is useful when you genuinely do not know what steps a request needs, but for the well understood, repeatable parts of your workflow, a static plan is easier to test, easier to explain to an auditor and cheaper to run since you are not paying for planning tokens on every call.
Designing for multi-tenant AI applications
For multi-tenant scenarios, the guidance mirrors standard SQL multi-tenancy practice: segment users by tenant and authenticate them into your application before any data reaches the model. The useful reminder here is that LLMs are stateless between calls. They do not retain or cache information from one request to the next on their own, so any cross-tenant leakage you see in testing is almost always a bug in your permission or data isolation logic, not the model deciding to share information it should not.
That is a helpful mental model when you are debugging a suspected leak. Check your retrieval filters and your vector store’s tenant scoping before you start suspecting the model itself.
Sharing a chat session between multiple users
The last question covers multi-user chat, where several employees want to join the same conversation with an AI assistant. The Copilot Chat sample app handles this the way a shared Word document works: once you share a chat with another user, they can see everything already in it, the same way sharing a document gives someone the same view you have.
That is a simple mental model to build against, but it is worth remembering that shared state introduces the same concurrency questions any collaborative app has. If two people send messages into the same session close together, you need to decide how the assistant’s memory and conversation history handle interleaved input, which the sample app does not fully solve for you out of the box.
What this Q&A tells you about early LLM app development
None of these answers involve exotic techniques. They are closer to good application architecture principles applied to a new kind of component. Authenticate before you authorize, keep the model away from raw data access, prefer fixed and testable execution paths where you can, and remember that the LLM itself holds no state between calls.
If you are starting a Semantic Kernel project today, treat this list as a checklist rather than trivia. Vector store choice, chunking strategy, SQL access pattern, planning strategy and tenant isolation are the five decisions that will determine whether your proof of concept survives contact with a security review.
Leave a Reply