In production, the fast-versus-deep model choice is the wrong axis. Classify the task, route it to the model that fits, and give every route a fallback chain.
Dennismary Chidiebere20 April 20265 min read
Operator Notes
Get the next essay in your inbox.
Field notes from building, deciding and carrying the cost of clarity.
Most AI systems I have seen pick their model the same way: a toggle between fast and deep. Cheap model for the quick stuff, expensive model for the hard stuff, and a prompt somewhere that tries to guess which bucket the request falls into.
It works in a demo. In production it quietly bleeds money and breaks at the worst time.
"Fast or deep" sounds like a model property. It is not. It is a property of the task, and it collapses two things that have nothing to do with each other: how much reasoning a task needs, and how much latency a user will tolerate.
Summarising a short document needs almost no reasoning and can be slow without anyone caring. Classifying an API response against a schema needs precision, runs thousands of times, and has to be cheap. Writing a verification plan needs real reasoning and happens once. Extracting a few fields from structured text is trivial and should never touch an expensive model.
Three of those are "fast" by a naive reading. They want completely different models. The moment you route on a single fast-or-deep flag, you either overpay for the trivial work or under-serve the hard work, and usually both at once.
So the axis is wrong. Stop asking how powerful the model is. Start asking what the task is.
Classify the task first
The fix is to name the task classes before you name the models. In Dino I settled on a small, closed set of task types (extraction, classification, short generation, long-form reasoning, planning, and a handful of others), each defined by what it demands rather than which model happens to serve it today.
A task class is a contract. It says: this kind of work needs this much reasoning, tolerates this much latency, runs at this volume, and is acceptable at this quality bar. Once that contract exists, choosing a model is a lookup, not a guess. Extraction maps to a small, fast model. Reasoning-heavy planning maps to a stronger one. Classification maps to whichever model is cheap and accurate enough at scale.
The payoff is that the mapping is explicit and reviewable. Anyone can read the table and see which model serves which task, and change one route without touching the rest. The intelligence moves out of a fragile prompt and into a decision you can see.
Every route needs a fallback chain
Here is the part people skip, and it is the part that matters at three in the morning.
Models go down. Providers rate-limit you. A region gets congested. A new version of a model starts refusing a prompt it used to accept. If a route points at exactly one model, then that model's bad day is your outage.
So no route points at one model. Each points at a chain: a primary, one or more fallbacks, and a defined failure at the end. If the primary is unavailable, the request falls through to the next model that can serve that task class, not a random substitute, but one chosen in advance to meet the same contract. Only when the whole chain is exhausted does the request fail, and it fails loudly with a clean error rather than hanging or silently returning worse work.
Two rules make this hold up. The fallback must be able to do the job: falling back from a reasoning model to one that cannot reason is not resilience, it is a quieter failure. And the chain must degrade gracefully and visibly: when you drop to a fallback, you log it, because a system that silently runs on its backup for a week is a system you no longer understand.
The structure is boring on purpose. A route is a task class, an ordered list of models, and a terminal error. That is the whole shape. Boring is what survives an outage.
Multi-tenant makes it unavoidable
If every tenant shared one model lineup, you could maybe get away with less. They do not. Enterprise customers want to bring their own keys and their own model preferences, which means the same task class resolves to different models for different tenants. And a tenant's own key, when present, takes priority over the platform default.
That only stays sane because the task class is the stable thing. A tenant expresses a preference against a class ("use my key for reasoning work"), not against a scattered set of call sites. The router reconciles tenant preference, platform default and availability into one ordered chain at request time. Preference first, then the platform's choice, then the fallbacks, then a clean failure.
Try to build that on a fast-or-deep toggle and you will drown in special cases. Build it on task classes and tenant preference is just one more input to the same lookup.
Measure per route or you are guessing
A router you cannot measure is a router you are superstitious about. For each route I want two numbers: what it costs and whether it is good enough.
Cost is the easy one: tokens and calls per task class, which immediately shows you the trivial, high-volume routes quietly spending the most money. Those are where a smaller model pays for itself fastest.
Quality is the one that takes discipline. Each task class has an acceptable-output definition, and you sample real traffic against it. When a route's quality drifts (a model update, a shift in input distribution), you see it on that route, not as a vague sense that "the AI got worse". Then you change one mapping, or reorder one chain, and you measure again.
That is the loop. Classify, route, fall back, measure, adjust one route. No heroics, no rewrites.
The fast-versus-deep toggle felt like a decision. It was really a refusal to make one. Name the task, and the model chooses itself.