Smarter SK Agents with Contextual Function Selection

Register enough plugins on a Semantic Kernel agent and you run into a problem that has nothing to do with your business logic. The model has to read through every function name, description, and parameter schema on every single turn, and once you cross two or three dozen functions, it starts picking the wrong one or missing the right one entirely.

Sergey Menshykh’s team on the Microsoft Agent Framework has now shipped Contextual Function Selection for the Semantic Kernel Agent Framework, a feature that narrows the function list the model sees, per turn, using vector similarity instead of handing over the entire catalog every time. It changes very little about how you write plugins and it changes almost everything about how large a plugin set you can realistically wire into one agent.

Why Large Function Sets Break Down

Picture a support agent wired to customer review retrieval, sentiment analysis, summarization, email and chat notifications, calendar lookups, and half a dozen Azure service calls. That is a realistic count for an enterprise assistant, easily 40 to 60 registered functions once you add every plugin a team wants bolted on.

Every one of those function signatures, along with its description and parameters, gets serialized into the model’s context on every call when you use FunctionChoiceBehavior.Auto() the standard way. Beyond roughly 20 to 30 functions, model providers start losing accuracy at telling similarly named or similarly described functions apart, and you end up paying input tokens for descriptions the model never calls.

What Contextual Function Selection Actually Does

The feature adds an AIContextProvider to the agent thread called ContextualFunctionProvider. On setup it embeds the description of every registered function into a vector store you choose. On each turn it embeds the current conversation context and runs a similarity search against that store, then hands the model only the top matches, controlled by a maxNumberOfFunctions setting, instead of the full list.

This is retrieval augmented generation applied to function metadata rather than to documents. The mechanics are the same: embed once, query per turn, retrieve the top results, inject them into the prompt. If you have already built a RAG pipeline for search or question answering, the pattern here will feel familiar.

Wiring It Up

The setup has four pieces: an embedding generator, the agent itself, the ContextualFunctionProvider registered on the agent thread, and a normal invoke call. Here is the C# from the review summarization sample the Semantic Kernel team shipped with the announcement.

// Step 1: create an embedding generator - this produces the vector used to compare
// function descriptions against the current conversation context
var embeddingGenerator = new AzureOpenAIClient(new Uri("<endpoint>"), new ApiKeyCredential("<api-key>"))
    .GetEmbeddingClient("<deployment-name>")
    .AsIEmbeddingGenerator();
 
// Step 2: create the agent as usual, with FunctionChoiceBehavior.Auto()
// so the model can still call whatever functions end up shortlisted
ChatCompletionAgent agent = new()
{
    Name = "ReviewGuru",
    Instructions = "You are a friendly assistant that summarizes key points " +
                   "and sentiments from customer reviews.",
    Kernel = kernel,
    Arguments = new(new PromptExecutionSettings
    {
        FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()
    })
};
 
// Step 3: register the ContextualFunctionProvider on the agent thread
agentThread.AIContextProviders.Add(
    new ContextualFunctionProvider(
        vectorStore: new InMemoryVectorStore(new InMemoryVectorStoreOptions
        {
            EmbeddingGenerator = embeddingGenerator
        }),
        vectorDimensions: 1536,             // must match the embedding model's output size
        functions: GetAvailableFunctions(),  // full universe: 16 functions in this sample
        maxNumberOfFunctions: 3              // shortlist size passed to the model each turn
    )
);
 
// Step 4: invoke normally - the provider runs its similarity search internally
// before the model ever sees the function list
ChatMessageContent message = await agent
    .InvokeAsync("Get and summarize customer review.", agentThread)
    .FirstAsync();
Console.WriteLine(message.Content);

The embedding generator is created once and reused, since it only does two jobs: embedding your function descriptions at registration time, and embedding the running conversation on every turn. vectorDimensions has to match whatever your embedding deployment actually returns, 1536 for text-embedding-ada-002 and text-embedding-3-small, 3072 if you switch to text-embedding-3-large without updating this number, and you will hit a runtime dimension mismatch instead of a compile error.

maxNumberOfFunctions is the whole point of the feature. Set it too low and the correct function for an edge-case query gets pushed out of the shortlist entirely, so the model cannot call it no matter how well it understands the request. Set it too high and you are back to the original overloaded-function-list problem, just with fewer entries.

What the Model Actually Sees

In the sample, GetAvailableFunctions() returns sixteen functions across six categories: customer reviews, sentiment analysis, summaries, communication, date and time, and Azure services. Asking the agent to fetch and summarize a review produces this output.

Customer Reviews:
-----------------
1. John D. - Rating: 5/5
   Comment: Great product and fast shipping!
   Date: 2023-10-01
 
2. Jane S. - Rating: 4/5
   Comment: Good quality, but delivery was a bit slow.
   Date: 2023-09-28
 
3. Mike J. - Rating: 3/5
   Comment: Average. Works as expected.
   Date: 2023-09-25
 
Summary:
--------
Overall sentiment is positive, with customers happy about product quality
and shipping speed. A few reviews point to delivery time as an area to watch.
 
Functions advertised to the model this turn:
----------------------------------------------
- Tools-GetCustomerReviews
- Tools-Summarize
- Tools-CollectSentiments

Only three of the sixteen registered functions made it into the model’s context for that turn: Tools-GetCustomerReviews, Tools-Summarize, and Tools-CollectSentiments. Every communication, date and time, or Azure service function stayed out of the prompt entirely, which is exactly the token and accuracy benefit the feature is built for. This selection happens before the model does any reasoning, so a badly phrased or ambiguous user message can produce a shortlist that misses the function you actually needed.

Where This Pays Off, and Where It Does Not

The feature earns its keep once you are past roughly 30 functions on a single kernel, especially when several of those functions have overlapping names or descriptions, the kind of thing that happens naturally once multiple teams contribute plugins to a shared agent. Below that count, the extra vector store dependency and the added embedding call on every turn are cost without much benefit, since current-generation models already handle a function list of that size reasonably well on their own.

Watch out for plugins whose descriptions genuinely look alike to an embedding model even though they behave very differently, several CRUD operations across different microservices are a common example. Semantic similarity search will happily shortlist three near-duplicate functions and leave the model just as confused as before, so in that situation explicit routing or better-differentiated descriptions will serve you better than relying on the provider to sort it out.

Production Considerations

InMemoryVectorStore is fine for a demo but resets on every process restart and does not share state across instances, so a production deployment behind a load balancer needs a persistent vector store, Azure AI Search, Qdrant, or one of the other connectors Semantic Kernel already supports, so every instance of your app searches the same function index.

The per-turn embedding call adds real latency, typically 50 to 150 milliseconds depending on your embedding deployment’s region and load, on top of the model call itself. For a chat experience that is usually acceptable, but for a latency-sensitive automation pipeline it is worth measuring before committing to the pattern.

Because the shortlist depends on embedding similarity, two nearly identical user messages can occasionally surface different functions, which makes agent behavior harder to unit test than a kernel with a fixed function list. Logging the selected function names on every turn is worth doing from day one, both for debugging and for building a dataset you can use to tune maxNumberOfFunctions later.

The Alternative Worth Knowing About

If you would rather not add a vector store to the mix, the older pattern of splitting one large kernel into several smaller kernels, each scoped to one domain, and routing the conversation to the right kernel with a lightweight intent classifier, solves the same overload problem without an embedding call on every turn. It costs more upfront design work mapping intents to kernels, but it stays fully deterministic, which matters if your agent’s behavior needs to be predictable for compliance or testing reasons.