I Was Paying Max Reasoning Effort for “What Port Does the Sidecar Use?”

Our internal docs assistant had one setting: reasoning effort high. Every request got the full chain of thought. The architecture questions deserved it. “Link me the on-call rota” did not. That was fine at pilot scale, and then we crossed 10,000 requests a week and I went looking at the traffic mix: 61% of requests were pure lookups. I couldn’t tell from the logs how much of their output was reasoning trace versus answer, only that the entire bucket was being priced at high effort.

So I built a router to sit in front of the model. It isn’t serving live traffic yet — every number below comes from replaying a week of real logs offline. Below is what I built, what the replay says it saves, and the routing rule that looked obviously correct and quietly under-routed the questions that mattered most.

What You’ll Build

A stateless effort router between your API gateway and Bedrock. It classifies each request into low / medium / high reasoning effort and injects the matching system prompt. No model call in the routing path: two compiled regexes, about 2 microseconds per request.

Replayed over 198 real requests, it cut output tokens 64.6% and cost from $9.26 to $3.33 per 10,000 requests.

Prerequisites and Stack

• Amazon Bedrock with openai.gpt-oss-120b-1:0 enabled ($0.15/1M input, $0.60/1M output)
• Python 3.12. The shipped router has no dependencies; tiktoken only for measuring token counts offline, which you’ll want.
• Somewhere stateless to run it. I used Lambda behind API Gateway.
• Real traffic. I pulled one week of gateway logs, sampled 198 requests to replay against, hand-labelled 60 more as a tuning set, and held back a further 40 that I left unlabelled until the router was frozen. Without labels I’d have tuned the threshold that turned out to be the broken one.

Architecture Overview

                     ┌──────────────────────────┐
  client ──────────► │  API Gateway             │
                     └────────────┬─────────────┘
                                  │
                     ┌────────────▼─────────────┐
                     │  Effort Router (Lambda)  │
                     │  ┌────────────────────┐  │
                     │  │ judgement verb?    ├──┼──► high
                     │  │ analysis verb?     ├──┼──► medium
                     │  │ else               ├──┼──► low
                     │  └────────────────────┘  │
                     └────────────┬─────────────┘
                                  │  system prompt:
                                  │  "Reasoning effort: {level}"
                     ┌────────────▼─────────────┐
                     │  Bedrock — gpt-oss-120b  │
                     └────────────┬─────────────┘
                                  │
                     ┌────────────▼─────────────┐
                     │  CloudWatch: effort +    │
                     │  output_tokens per req   │
                     └──────────────────────────┘

The router never calls a model. A classifier LLM in the hot path costs you a network hop plus its own tokens, which is most of what you came to save. Every decision here is string matching on a request that’s already in memory.

Step-by-Step Walkthrough

1. Confirm your model honours effort via system prompt. With gpt-oss this is documented behaviour: the chat template prepends Reasoning effort: low|medium|high to the system message. Same weights and the same endpoint; only the trace length changes. Send one prompt at all three levels and diff the output token counts before you build anything on top of it.

2. Measure your actual effort distribution. I bucketed the 198 sampled requests by hand:

3. Write the router as pure functions.

import re
HIGH = re.compile(r"\b(can we|should we|safe to|actually|break\w*|broken|"
                  r"guarantee\w*|exactly-once|surviv\w*|design|prove|reconcile|"
                  r"trade-?offs?|failure modes?|allocate|budget)\b", re.I)
MED  = re.compile(r"\b(why|compares?|explains?|walk me through|summar\w*|drafts?|"
                  r"difference between|what (would |will )?happens? if)\b", re.I)
def route(q: str) -> str:
    if HIGH.search(q): return "high"
    if MED.search(q):  return "medium"
    return "low"

Two details worth stealing. Match stems rather than exact forms: guarantee\w* catches guaranteed, and break\w* plus broken catches the inflections people actually type. Bare guarantees? misses “is ordering guaranteed here,” which is precisely the kind of short question you cannot afford to under-route. Also keep HIGH first, because one question can match both patterns. Why can we drop the lock hits why and can we, and it has to resolve upward.

HIGH also carries consequence nouns (exactly-once, failure modes, trade-offs) and one adverb, actually, which turned out to be the single highest-signal token in the set. Nobody says “is it actually X” about something they don’t already suspect is wrong.

4. Inject and log. Prepend the effort line to the system message, then emit effort and output_tokens as CloudWatch dimensions on every call. Without that second dimension you can watch the bill fall without knowing which bucket moved.

5. Freeze the router, then label the held-out set. This ordering matters and I got it wrong the first time.

What Broke

My first router had a rule I was sure of: under 18 tokens goes to low, over 45 goes to high. Prompt length as a proxy for difficulty.

On the tuning set it misrouted 10 of 60, split evenly between over- and under-routing. The over-routes were harmless: someone pastes a 60-token stack trace, asks “which runbook covers this,” and we pay for a full reasoning trace on a lookup.

Then I labelled the held-out 40, and the split stopped being even. All 13 misroutes there were under-routes. These landed on low:

• “is our outbox pattern actually exactly-once”
• “can we drop the distributed lock here”
• “does backpressure break our sla”
• “should we shard the orders table”

Every one is short, and every one is a load-bearing architectural judgement where a fast, confident, wrong answer is worse than no answer at all. The length heuristic had it backwards. In a system you already understand, you’ve compressed the context out of the question before you ask it, so the hardest questions arrive shortest.

I deleted the length rule and replaced it with the modal and consequence verbs in the listing above, which force high regardless of length.

Validation and Results

Same 198 requests, replayed offline against all three configurations and priced at Bedrock’s published gpt-oss-120b rates. The whole replay cost 31 cents.

Cost per 10k is the 198-request replay scaled linearly, holding input at the measured 53.6 tokens per request. The verb rule gives up 1.1 percentage points of savings and removes every under-route I can find. That’s the trade, and it isn’t close.

One caveat on the zeros: I wrote both label sets myself, so read 0/40 as “no known failures,” not as an error rate. route() benchmarked at 2.1 µs median over 10,000 calls, which is noise against a Bedrock round trip.

Key Takeaways

• Effort is a system prompt on one set of weights. One model, one endpoint, no extra deploy surface — the architecture cost is roughly zero.
• Route on verb, not length. Modal and consequence verbs are the signal that a question needs reasoning.
• Over-routing wastes money and under-routing ships wrong answers. Those aren’t symmetric, so bias the router toward spending.
• Label a held-out set after freezing the router. My v1 looked survivable at 10/60 on the set I tuned against and fell apart at 13/40 on data it hadn’t seen.

What’s Next

Two things I’m testing. First, a small embedding classifier trained on the labelled set, to find out whether it beats a dozen lines of regex by enough to justify a model in the path. The regex is phrasing-brittle in a way that will not survive contact with new users, and stem-matching only papers over so much of that. Second, one-way escalation: let a low response that trips a hedging detector retry once at high, and check whether the retry cost stays under the routing savings.

Try It on Your Own Logs

If you’re running a single effort level across mixed traffic, pull last week’s logs, bucket 60 requests by hand, and price the tail. The number will probably annoy you. And if you build a version of this, tell me what your held-out misroute rate came out at. I want to know whether the short-hard-question failure mode is universal or just mine.

Prompted by Sebastian Raschka’s “Controlling Reasoning Effort in LLMs” (Ahead of AI, 18 Jul 2026) and ByteByteGo’s “How ChatGPT Optimizes its Agent Loop” (29 Jul 2026).