We designed a difficulty router three times, built one, and shipped none of them.

My impression, and I cannot back it with numbers, is that auto routing is what people bring up with me most often, in Discord and on calls. Our issue tracker will not back it up either, so treat it as impression rather than data. Token routing, smart routing, auto router, pick your name. Nearly every gateway is shipping something called routing, though a lot of it is routing you configure yourself rather than routing that picks the model for you. We have not shipped one that judges how hard your request is. What we do ship is an example routing hook that picks a model from the shape of a request, tools declared, output size, message count, scored against measured latency and a declared cost table. That is the policy I wrote about in July, and it is in the repo. The thing I could not justify shipping is the layer above it, the one that reads the request and decides how hard it is.

The honest answer is that we tried three times and nothing survived intact. The first died on the published research. The second survived as arithmetic and killed the product, though it does not hold for every pairing, as I show below. The third lost the detector at its centre when two reviewers ran it and it fired backwards, and what is left of that attempt is the only version of this I am still willing to build. What I cannot claim is that the work is finished. Our cache analyser has never been run against a corpus, and the cheapest experiment we identified, whether escalation has any traffic to act on at all, we never ran either. I would rather publish that than ship something that demos well and quietly costs people money. To be clear about the scope of the claim: this is not me saying nobody’s router works, it is me showing what we could and could not justify, with the arithmetic in the open, so that anyone who has solved it can tell me where I went wrong.

I still think the argument in that post holds, that the smart router you want is a hook. What changed is that we went and tried to write the policy people actually ask for, the one that judges difficulty, and that is where it fell apart.

Attempt one: classify the prompt, route to a cheaper model

This is the design behind most of the routers that genuinely pick a model for you, though it is a minority of what gateways call routing. Look at the incoming request, judge how hard it is, send the easy ones to a small model and the hard ones to a big one. Amazon Bedrock’s intelligent prompt routing “analyzes the prompt to understand its content and context” and picks between two models in the same family. OpenRouter’s Auto Beta router runs a lightweight classifier over about thirty task types and then ranks by the community’s trailing seven day share of spend.

We got a long way into designing this before going properly through the published research, which was my mistake and worth doing first.

The published results are more mixed than the launch posts suggest, and the limits come from both sides of the argument. The RouteLLM authors are candid about where their own routers stop generalising, an independent study of 21 routing methods finds them converging on a plateau far below the oracle, and the single piece of evidence I found hardest to argue with is a router vendor switching its own router off.

RouteLLM, the canonical open router, is candid in its own ICLR 2025 paper about where its routers stop generalising:

On MMLU (Table 2), all routers perform poorly at the level of the random router when trained only on Arena dataset, which we attribute to most MMLU questions being out-of-distribution (see Section 5.3).

That result is about MMLU specifically, and about training on Chatbot Arena data alone. GSM8K does not rescue it either. The paper reports that “similar to MMLU, the performance of all routers trained only on the Arena dataset is close to random”, with an APGR below random until the training set is augmented. The paper’s own fix sits in the same tables. On GSM8K, adding synthetic data from an LLM judge takes all routers, in their words, “from an APGR worse than random to an APGR greater than random”, and the best of them, the causal LLM classifier, ends up “requiring 17% less GPT-4 calls than random to achieve CPT(50%) and CPT(80%)”. On MMLU the fix is different data and a different number: golden labels from the validation split leave all routers “requiring approximately 20% less GPT-4 calls than random for CPT(50%)”. Their abstract’s own headline is that this reduces costs “by over 2 times in certain cases” without compromising response quality. Their best single result, MT Bench at CPT(50%), reaches 3.66x, against 1.41x on MMLU and 1.49x on GSM8K. So this is not a paper saying routing never works. It is a paper saying a router is only as good as the match between its training distribution and your traffic. Which, if your traffic is agentic coding and the training data is chat preferences, is the case you are in.

RouterEval (EMNLP 2025 Findings) built a benchmark from over 200 million performance records across more than 8,500 models. I want to represent them fairly, because their conclusion is optimistic: they find that a capable router’s improvement “can even surpass the performance of the best single model in the pool”, and they call routing “a highly promising paradigm”.

The part I take from it is the gap between that ceiling and what existing methods reach. The routers that clear the best single model in their scaling curves are constructed from the answer key: they build an oracle router out of the recorded results and blend it with a random sampler to simulate routers of varying capability. When they evaluate methods that actually exist, their abstract ends its findings with this:

Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement.

So the ceiling is real and nobody is close to it. That gap matters. The claim gets repeated as “routers cannot beat the best model in their pool,” and that is not what the paper says. What it says is that the theoretical headroom is large and current methods are not capturing it, which is a reason to keep researching and a poor reason to ship one today.

The Routing Plateau named the pattern:

Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau: many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router.

They put numbers on both halves of that. On the narrowness: “on RouterBench, the top 15 routers differ by only 0.23 percentage points (pp) in accuracy”. On the gap to perfect routing: they find the best router trailing the oracle by 10 to 30 percentage points across the five benchmarks. A whole field of methods packed into a quarter of a point, all of it well under what an oracle would get.

Their diagnosis is the interesting part, and they call it a predictability bottleneck: current routers “mainly learn global averaged model-performance trends rather than fine-grained query-specific routing signals”, so they “solve overlapping easy queries but collectively fail on hard queries that require instance-specific routing decisions”. In other words the routers agree with each other because they have all learned roughly the same thing, which is which model is generally better, not which model is better for this query. That is the whole game, and it is the part that is not working.

To be fair to them, they do not conclude it is hopeless: they find larger training sets, stronger encoders and end-to-end fine-tuning move the needle. It is a plateau, not a wall.

Then there is the structural argument, and it is the one that actually changed how I think about this. SWE-Router, which looks at routing in multi-turn agentic software tasks, frames prompt routing as an information problem rather than a modelling problem. In their words, existing routers “operate on the task description alone, which inherits an information-theoretic Bayes-error floor in agentic settings: a similar issue can hide either a localized typo or a multi-module refactor, and the prompt does not separate the two.” If the information is not in the input at decision time, no classifier recovers it. A better classifier does not move a Bayes floor.

I should be straight about where that paper goes next, because it is not where I went. Their answer is to change the input rather than the classifier: let the cheap model run a few exploratory turns and read the partial trajectory before deciding. They prove that conditioning on the trajectory “never harms routing and is strictly better whenever exploration is informative”, and report that SWE-Router “greatly improves the cost efficiency of SWE tasks, while maintaining the majority of the performances of the stronger model”. So it is not a paper saying give up on routing. It is a paper saying give up on routing from the prompt, and it is the strongest published argument I found for the thing I try in attempt three.

But the piece of evidence I keep coming back to is not a paper at all. It is Manifest, on why they deprecated their own LLM router, and if you read one link from this post, read that one.

They are not theorising, and they are not bystanders. Manifest sells an open source LLM gateway, and their words are that they “launched the Manifest LLM router in March as a key feature in our LLM gateway”. The router’s whole job was tier classification: every request sorted into simple, standard, complex or reasoning. They ran it in production for four months across 7000 cloud users, deprecated it in June, and it shuts down for good on September 1. The gateway stays. The part that picks the model for you is the part that is going. Their explanation is the sentence I could not get past:

The prompt alone does not contain the whole task; it is just the trigger.

And on whether it saved anything:

The amount saved is paid somewhere else, and that cost is harder to estimate.

That is a team that shipped the feature, ran it in production for four months, and turned it off. I should be straight that they did not arrive from the other direction and they did not arrive second. Their first section is that complexity cannot be deduced from the prompt alone, which is my attempt one. Their second is that cache reads are 75 to 90 percent cheaper than uncached inputs, so a cache aware router ends up doing its job by not doing it, which is my attempt two. They published it before I wrote this. And what they report is field experience rather than a measurement: their own words are “mixed results and a lot of GitHub issues and discussions about it”, “based on our experience”, and that the displaced cost “is harder to estimate”.

I want to say something less analytical about that post, because it is the reason this one exists. For weeks I had been reading launch after launch, watching people I respect swear by auto routing, and quietly assuming I was the one missing something. That is a bad place to make decisions from. Reading Bruno Perez describe shipping the thing, running it in production for four months, and then deprecating it was the first moment I felt sane rather than slow. So, Bruno: thank you. Retiring a shipped feature in public, at a company that sells a gateway with a router in it, is rarer than it should be, and it is worth more to the rest of us than another launch post.

Attempt two: the arithmetic, which is the part I would most like to be wrong about

Set aside whether you can classify correctly. Assume a perfect oracle that always knows the right tier. Does moving a turn to a cheaper model actually save money in an agent session?

Work it through with published list prices. I am using one vendor’s lineup because it is the one our own traffic runs on, not because I have measured what most coding agents use.

At the prices published as I write this, Claude Opus 5 is $5.00 per million input tokens, and a cache read is a tenth of that, so $0.50. Claude Haiku 4.5 is $1.00 per million input tokens.

Read those two numbers again, because they are the whole post.

Per token, a cached turn on Opus 5 costs half as much on input as an uncached turn on Haiku 4.5. Prompt caches are per model, so the first turn you route down reads the entire context cold. On that turn the downshift does not save a cent on input, it costs more. The direction of the saving is backwards.

That leaves output. On the standard rate card the gap is a flat five times across input, output, cache write and cache read alike, but output is the only place it survives a cold cache read: $25.00 versus $5.00 per million. Five times, not the fifty times that separates an Opus 5 cache read from an Opus 5 output token. So for one routed turn, downshifting Opus 5 to Haiku 4.5, the break even is roughly:

output_tokens_needed = prefix_tokens / 40

A 100k token context needs about 2,500 output tokens before that downshifted turn pays for itself. Now think about what a coding turn looks like. Large context, small output. I do not have a published distribution for our own traffic, and I have not found one published for agentic coding turns anywhere, so treat this as impression and not measurement, but most of the turns I watch answer a question, patch a file or call a tool in a few hundred tokens, and turns like that sit below the line. The threshold is the honest part of that sentence rather than the impression: if your turns routinely emit 3,000 output tokens on a 100k context, you are above the line and this argument does not apply to you.

That divisor of 40 charges the cold prefix on the cheap model at its input rate. If it bills as a cache write instead, at $1.25 per million on Haiku 4.5, the divisor is about 27 and a 100k context needs roughly 3,700 output tokens. I have left 40 in the formula because it is the version least favourable to my own argument, but 27 is the one to use if you are checking whether a real switch pays.

That is also where thinking cuts against me, and it is worth saying plainly rather than burying it. Reasoning tokens bill as output. On Opus 5 the manual budget form of extended thinking is not even available, since Anthropic’s docs say Claude 4.7 and later models “reject requests that use it, returning a 400 error”, and adaptive thinking is on instead, with the docs stating that on Opus 5 “thinking is already on: no configuration needed”. So this is the default state of the model the whole section is priced on, not an opt in. A turn that thinks at length can clear 2,500 output tokens on its own, which is exactly the regime where my own formula says the downshift pays. The more your traffic thinks, the more the arithmetic moves toward routing, not away from it.

Four more things have to be said about that formula, because it is narrower than it looks and I would rather say them than have them said to me.

It is specific to that pair, Opus 5 down to Haiku 4.5. Downshift from Opus 5 to Sonnet 5 instead and the divisor is 10 at today’s introductory pricing, and 4 once the standard $3 and $15 pricing starts on September 1. Go the other way, to the top of the lineup, and the argument inverts completely. Anthropic’s docs call Claude Fable 5 their most capable widely released model, and a Fable 5 cache read is $1.00 per million, exactly the same as a cold Haiku 4.5 read. Input is a wash, the output gap is $50 to $5, and the downshift pays on any turn at all.

It is specific to a one-off switch. Stay on Haiku 4.5 and it builds its own cache, which reads at $0.10 per million against Opus 5’s $0.50. You pay a $1.25 per million write to get there, so the cumulative input cost turns positive around the third downshifted turn.

What that trades against is the trip back. If the detour outlasts the five minute cache window, re-entering Opus 5 costs a $6.25 per million cache write instead of a $0.50 read. On a 100k prefix that is about 58 cents more than the read it replaces. Against that, the first downshifted turn actually loses about 4.5 cents, because the cold Haiku write costs more than the Opus read it replaces, and every turn after that nets about 7 cents, 4 cents of input once Haiku is reading its own cache plus 3 cents of output on a turn that emits 1,500 tokens. A round trip needs about ten turns on the cheap model before it is worth taking, and per-turn routing never gets that.

And one detail cuts against me. Opus 5 uses the tokenizer introduced with Opus 4.7, which Anthropic says produces roughly 30% more tokens for the same text than the one Haiku 4.5 uses. The same transcript is therefore cheaper in Haiku tokens than the formula above assumes, and the true break even is lower than prefix over 40. I do not think that reverses the picture for short-output turns, but it narrows it, and if you are checking my numbers this is the first place to push.

There is a real insight buried in that formula and I will give it away, because I would rather somebody built it than nobody did. The break even depends on the ratio of output to context, not on difficulty. So the useful question was never “how hard is this turn.” It is “how much output will this turn produce relative to its context.” Long generation turns can pay. Short answers on a big context never do, no matter how trivial they are. I have not found a router that keys on that ratio. The closest I found is R2-Router, which treats the output length budget as a controllable variable and picks the model and the budget together, but it holds input length fixed and never uses the ratio. If there is a closer one, tell me.

If your numbers say otherwise, I want to see them. Different vendor mixes and different traffic shapes will move this, and self hosted fleets change it completely because you control the cache.

Attempt three: stop predicting, start watching

If you cannot tell from the prompt, watch the work instead. Let the cheap model start, notice when it is going badly, escalate. Route up rather than down. This has better evidence behind it. SWE-Router is built on exactly this idea, and it proves the point structurally: conditioning the routing decision on a partial trajectory “never harms routing and is strictly better whenever exploration is informative”. It is not free, though, and the paper is candid about that. The cheap model’s turns are “paid up-front and counted in any escalated run’s cost”, on escalation the strong model “restarts from q” rather than inheriting the cheap model’s context, and the result maintains “the majority of the performances of the stronger model” rather than all of it. So escalation buys a better decision, not a free one. You pay for the exploration and you eat the same cold cache read I just used to argue against routing down. What it has over prompt classification is that it decides on evidence instead of on a guess.

It also targets the thing that actually costs money in agent work, which is not the price of a turn but the number of turns. Catching a model going in circles on turn three instead of turn seven removes four turns, and it escalates against a smaller context than a turn seven escalation would, because the cold read the expensive model pays scales with however far the loop was allowed to run. That is a different axis from shaving a percentage off one request. The general purpose routing benchmarks I leaned on above score one routing decision per query and label it in isolation, so turn count is invisible to them even where the source data is a conversation. RouterEval leaves cost out entirely and ranks on accuracy. RouteLLM measures the share of calls sent to the strong model, and it scores MT Bench as “160 open-ended questions using LLM-as-a-judge”, one routing decision per question. The Routing Plateau does attach a per query and model cost, on four of its five benchmarks, but that suite includes RouterBench, which files MT-Bench under conversation and then, in its own words, “only used the 80 questions”. None of them scores how many turns a run took, so nothing in the metric rewards a run that finishes in fewer turns. SWE-Router is the exception, and it is the reason I take the point seriously: it routes on the trajectory rather than the prompt, and it releases a multi-LLM trajectory dataset so trajectory-level routing can be reproduced.

So we designed a detector: watch how the model writes, catch the hedging and the repetition that show up when something is stuck.

Then two reviewers ran it, unmodified, against constructed transcript shapes, and it was worse than useless. It fired on an eight crate rename that was going perfectly, on a green test suite, and on a user simply repeating an instruction. On an opening bug report, before any work had started, it scored two of the three points it needs to escalate. It stayed quiet on a model narrating genuine repeated failure in fresh words. The signal was not weak, it was inverted. The premise, that a model making progress writes something new each turn, breaks down whenever the work itself is repetitive, and repetitive work turned up constantly in the shapes we tried. I have not counted what share of real agent traffic looks like that, so take it as our experience and not a statistic.

The reason two people could falsify it by running it, instead of arguing about it, is that the design shipped with working code. Nobody timed them, so I am not going to put a number on how long it took. That is now a rule for us. These are internal results with no public artifact you can check, which is a lower standard than I would accept from anyone else, so weigh them accordingly.

There is a second reason it cannot be tuned back into shape. A gateway sees thinking blocks, but not the reasoning inside them. On the newest models the display setting defaults to omitted, so the thinking field comes back empty with an encrypted signature, and even when a caller opts into summaries the vendor’s own docs say no setting returns the raw chain of thought and that the summary is written by a different model from the one you called. So the stream a distress detector would point at is either absent or a paraphrase. And where you can see reasoning text, hedging and backtracking look to me like the ordinary texture of a reasoning trace rather than a symptom. A distress detector pointed at that stream is measuring the weather.

Why we do not think this is feasible

Take the turn that settled it for me. Somebody types “yes, do that.”

Three words. Every difficulty classifier I have designed or read about grades that simple, and grades it simple correctly, because as a piece of text it is about as simple as text gets. So it routes to the cheap model. But “yes, do that” is not the task. It is a signature on a task described somewhere else, possibly several turns ago, possibly at length. What follows might be renaming a variable, or it might be the eight crate refactor the assistant proposed two turns back. Nothing in those three words separates them.

There are only two questions worth asking about that, and I could not answer either one well enough to ship.

How would you route it correctly? The only way to know what “yes, do that” authorises is to read the conversation that came before it. Not the last message, the conversation. And that is not a tweak to a prompt classifier, it is a change of object: you are no longer grading the request, you are grading the history that gives the request its meaning. Everything follows from that swap. How far back do you read. What do you do when the turn that matters has fallen outside the window. How do you weigh what the assistant proposed against what the human actually approved, given the human may have approved only part of it. Each of those is a judgement call with no ground truth to check it against, which is the same labelling problem that killed our detector in attempt three.

And to what effect? Suppose you build it anyway. You are now paying a model call, on every turn, to classify a context that grows on every turn, in order to re-derive something the previous turn already established. The classifier can be the cheapest model you can get away with, but it is not free, and the thing it has to read is the thing that grows without bound. That cost scales with exactly the traffic the router is supposed to be saving money on. Set it against the arithmetic in attempt two, where the saving on a downshifted turn is a few cents and sits on the wrong side of zero whenever output is short relative to context, which is most agentic turns. You are adding a per turn cost that grows with the conversation in order to chase a per turn saving that does not.

And after all of it, you have classified what was said, not what the model is about to do. That is the part I keep coming back to, because it is the part no amount of engineering gets you out of. The work is in the doing. A short approval can authorise an afternoon of edits, and nothing in the text of the approval tells you that. A conversation classifier could resolve “yes, do that” perfectly, identify exactly which proposal it approves, and still be guessing, because the difficulty of a piece of agent work is a property of the work rather than of the sentence authorising it. The same approved plan is easy in a clean repository and hard in one where the tests are flaky and the types do not line up, and none of that is in the transcript at decision time. That is the Bayes floor SWE-Router names, reached from the other end: if the information is not in the input, a better classifier does not recover it.

Manifest put the whole of that in one line, and it is the sentence I could not get past:

The prompt alone does not contain the whole task; it is just the trigger.

They earned that sentence the expensive way, by shipping a router, running it for four months, and retiring it in public. It is the conclusion they reached from operating one, and it is where the arithmetic and our own failed detector arrived from the other two directions. Everything above is me working out on paper why it is true.

None of this proves the thing cannot be built, and I would not want it read that way. It says we could not justify building it, and the shape of that failure is what I would want someone to argue with. We did not stall on engineering. We stalled on being unable to say what the classifier would be measuring, or what it would save if it measured it correctly. When those are still the open questions, shipping it behind a flag and hoping the numbers come good later is not a plan.

What we think is actually true

• Prompt caching is real, measurable, and boring. It wins on arithmetic you can check: a cache read on Opus 5 is $0.50 per million against $5.00 uncached, a published tenth. Prompt compression is a different kind of bet, because it changes the prompt rather than repricing it, so its case rests on published evals rather than a price table.
• Escalating on evidence beats predicting from text, and routing up is where the saving in agent work lives, because turns are the unit of cost.
• The one signal in a conversation that is not a guess is the human. When a person spends a turn saying “that is still wrong,” they have told you the last attempt failed. That is an observation. It needs no classifier and no model of difficulty.
• Turns to completion is measured, but it is rarely the ranking metric. SWE-agent reports turns to resolution in Appendix B.3.1, TheAgentCompany’s headline results table carries Steps alongside Success, Score and Costs. It is treated as a footnote to accuracy. For agent economics it is the headline.
• The most promising lever may not be routing at all. If the prompt does not contain the whole task, the direct answer is to SUPPLY the missing context rather than try to infer it. That is what agent memory does, and it attacks the turn-count axis head on: a model that does not have to rediscover what you told it last week finishes in fewer turns. We use Hindsight for exactly this, every day, and it has quietly become one of those tools we would not want to work without. That is experience rather than measurement, and the distinction matters given everything else in this post, so I am not going to put a number on it. Their site leads with a state-of-the-art benchmark claim, and I have not checked it, which is exactly the kind of thing this post says you should not take on trust. But the effect is not subtle: not re-explaining the same context is worth more than shaving cents off a turn. It is the next thing we intend to put numbers on, the same way we did here, and if it holds up it is a far better candidate for a busbar plugin than any router we designed.
• If routing saves you less than the noise in your own traffic, the honest answer is to not ship a router for that segment.

Where we might be wrong, and what would change our minds

This is the actual ask. I have no interest in being the person who explains why a popular thing cannot work and turns out to have missed something obvious.

Specifically, I would like to hear from you if:

1. You have measured router savings against a realistic baseline, meaning what you would otherwise have configured, not the most expensive option, and it is still meaningfully positive. Especially on multi turn agent traffic rather than chat.
2. You are routing across vendors where the price spread is much wider than within one family. Our arithmetic is a within-family argument, and it may simply not survive contact with a mixed fleet. I suspect this is the strongest counterargument.
3. You run a self hosted or hybrid fleet. When you own the KV cache, cache aware routing is genuinely solvable, and the inference serving world has been doing prefix aware scheduling for a while. We may be reasoning too much from hosted API constraints.
4. You have a working flail or stuck detector. We could not build one from text. If you have, I would like to know what signal you used and how you got ground truth for “this was going badly,” because that labelling problem defeated us before the detector did.
5. You think the difficulty classification result is wrong. I am not claiming no router has ever beaten the best model in its pool. RouterEval’s own tables have cells where an existing method does, on MMLU and on GPQA at pool size three, and their results section is careful to scope the negative finding to “most settings” rather than to all of them. What I have not seen is a router that beats the best single model in its pool on your own production traffic, at lower cost, against the baseline you would otherwise have configured. That is the reply I would most like to get.

We are still going to build something here. The current plan is to instrument first and act second: ship the measurement, publish what the traffic actually says, and only add a decision once there is something real to decide on. If that produces a null result, we will publish that too.

Busbar’s job is to be the layer that does not lie to you about your traffic. Shipping a savings number computed against a baseline nobody would have run would be the opposite of that.

Tell me what we missed. Reply here, or open a discussion on the Busbar repo.

Prices quoted are published list prices as of the date on this post and change often. The research is linked from the papers named above; if I have characterised any of it unfairly, tell me and I will correct this post rather than quietly edit it.

References

• Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, ICLR 2025.
• RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs, EMNLP 2025 Findings.
• The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers.
• SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks.
• Manifest, Everyone is building LLM routers, we deprecated ours.

If I have read any of these uncharitably, say so and I will correct the post in place with a note rather than quietly editing it.