We asked the wrong question for longer than I'd like to admit.
"Which model is best for coding?" I've said some version of that probably a hundred times. In
planning docs. In conversations with engineers. In the reasoning behind which model we wired into
the pipeline first. It felt like the right question. You evaluate models, you pick the best one,
you build on top of it.
The data says the question itself is wrong. At simple and moderate complexity, two models we
tested were identical. At higher complexity, one held while the other started leaving routes out
about 30% of the time. Neither model is "better." They fail at different points on the complexity
curve in ways that benchmarks don't surface.
That's not a model quality story. It's an infrastructure story. If you're running a single model
for everything, you're either over-provisioning on simple tasks, under-provisioning on complex ones,
or both — depending on the day.
The complexity ladder
Before you can route to the right model, you need a way to describe what you're routing.
We use a measure called MoC — Measurement of Complexity — that scores a task specification before
any model touches it. It reads the spec's declared scope: how many files change, how many API
surfaces are involved, how many data model changes are required. The output is a t-shirt bucket:
XS, S, M, L, XL.
Think of MoC as the DNA of a task. It's intrinsic — it describes what the task is, not
how a model performed on it. A single health endpoint has different DNA than four routes including a
stateful reset. You can score the spec and know what class of task you're dealing with before you
dispatch a single inference call.
We built a ladder of specs with rising MoC — XS (one route), S (two routes), M (four
routes including a POST that calls existing state) — and ran aider in diff mode against two models
under faithful context (real spec + full project ledger, mirroring how our agentic coding pipeline
runs in production), ten trials per cell.
| Rung (MoC) |
qwen3-coder |
gpt-oss |
| XS: /health |
10/10 |
10/10 |
| S: /health + /version |
10/10 |
10/10 |
| M: /health + /version + /status + POST /reset |
7/10 |
10/10 |
N=10 per cell · 0 truncation events across 60 runs
gpt-oss is clean across the whole ladder. qwen3-coder's drop at M is a mild completeness fray —
it occasionally leaves one of the four routes out, not a truncation event. The zero-truncation
result is the definitive finding from this run. aider in diff mode never gutted a file across sixty
trials at three complexity levels. That's structural — surgical edit format cannot drop existing code
because it never re-emits the file body. The safety question is closed. Where each model's
completeness rate eventually falls off is still being measured; L and XL rungs need a multi-file
fixture. But that's a capability question, not a safety one. The routing layer can work with an
imperfect model at M complexity. It can't work with one that silently destroys files.
The first broker attempt — and why we threw it out
The obvious next step: build something that reads this data and picks the right model
automatically. We did. We called it the inference broker stub. It seeded a hand-authored dictionary
from the lab results, ranked models across four named axes (privacy, speed, cost, quality), and
resolved a "right model" at dispatch time.
We threw it out before it shipped. Four specific problems, each verified in code:
1. The data source was wrong. It sourced a static hand-authored dictionary from
lab results. By the time it was being reviewed, the dictionary had already diverged from production
— the model we'd validated as the top pick for agentic coding tasks wasn't in the index. A static
dict rots on day one.
2. It recommended models that can't run. The index ranked several engines as top
picks. Our production pipeline supports three execution modes. The index's headline pick couldn't run
in any of them.
3. MoC was decorative. The resolve algorithm accepted a MoC bucket as input but
never used it in ranking. The complexity signal that motivated building the broker in the first place
was silently ignored in the implementation.
4. Privacy was in the wrong place. Privacy was modeled as a sort key —
something you weigh against quality and speed. That's the wrong frame. Privacy is a gate: either a
model runs on operator-controlled infrastructure or it doesn't. Treating it as a dial means a "high
enough quality" model could route around a privacy requirement. That's not acceptable.
Every one of these was a design error, not an implementation bug. The spec itself was wrong. The
right response was to redesign from the ground up, not patch.
What the right architecture looks like
The insight from throwing out the static broker: you already have the data. Every pipeline
execution already writes a labeled data point — which model, which task, outcome, quality score,
latency, tokens. The right routing layer doesn't need a hand-authored dictionary. It reads the
matrix that production builds for you.
The architecture we're building works in four phases:
Measure. Stamp every agent execution with the task's MoC bucket and the engine
used. This is already running in production. Every run now writes
(agent, model, engine, moc_bucket, outcome, quality_score, latency, tokens) to the
execution log. The matrix exists. It self-updates.
Gate. Before ranking anything, eliminate candidates that can't run. Privacy is
a hard gate — a model whose endpoint doesn't meet the project's privacy policy is out, regardless
of quality. Context window fit is a gate. Tool-call compliance is a gate for agentic tasks.
Availability is a gate. Gates are binary. Nothing trades off against them.
Rank. Within the candidates that pass all gates, rank by the project's declared
preferences: quality (quality score + acceptance rate), speed (median latency), cost (real token
spend), consistency (variance across runs), reliability (error and timeout rate). The weights are
the project's policy — an operator who cares most about quality-first gets a different ranked ladder
than one who optimizes for speed. The ranking comes from production telemetry, not lab benchmarks.
Escalate. First-pick fails? Dispatch falls to the next rung on the ladder. The
fallback chain is dynamically ranked, not statically configured in a config file. If the telemetry
shifts — a model starts degrading, a new one gets validated — the ladder updates.
The matrix is already running. Every production execution is stamping a data point — model, task
class, complexity, outcome, latency, cost. The routing decision engine reads from that matrix when
it ships; the signal it needs is accumulating now. The flywheel is turning before the router
exists.
The governance problem hiding in plain sight
The routing layer is the visible part. The less visible part is what makes it possible at all:
per-execution attribution.
If you can't answer "which model ran which task, at what complexity, in how long, with what
outcome" — you're flying blind. Not in a vague sense. In the specific sense that when something goes
wrong, you can't trace it. When costs spike, you can't attribute them. When a model starts
degrading, you won't know until the damage accumulates.
This is the governance problem hiding in the local AI stack. It's not about security or
compliance in the traditional sense. It's about basic operational visibility: can you look at your
own system and understand what it's doing?
An engine that times out at 1200 seconds on a task that aider:diff clears in 21 seconds on a
different model isn't a model problem — it's a tooling problem. Without engine and model logged per
execution, that burn shows up as "infrastructure costs are higher than expected" with no trace to
the specific combination that caused it. The telemetry isn't an add-on — it's the foundation that
makes routing possible and cost accountability legible.
The MoC score, the execution log, the routing layer — all three are load-bearing. Score the task,
log the run, route on the evidence. Without any one of them, the other two don't work.
Where this is going
The matrix the routing layer reads from is already accumulating data from production runs. As more
models run against more task classes, the signal improves. When a new model appears or an existing
one starts changing behavior, the matrix reflects it without a manual update.
The next phase is offline validation: run the routing algorithm against the production matrix,
compare what the routing layer would have picked against what operators actually dispatched, measure
non-regression. When it clears that bar, it ships in shadow mode — logging picks without acting on
them — before it routes live work.
"Which model is best?" is the wrong question because it treats model selection as a one-time
decision. The right infrastructure makes it a continuous measurement. That measurement is already
running.
Three-part series from QCoda Labs. Episode 1: silent tool-call failures on raw Ollama
endpoints — the serving layer, not the model. Episode 2: engine × model compatibility — edit format
discipline mattered more than parameter count. Episode 3 (this post): task complexity,
per-execution attribution, and why the routing layer is infrastructure, not a feature.
