We expected the best model to win.
That's the intuition, right? Bigger parameter count. Better HumanEval score. Coder-tuned on more
data. You pick the strongest model you can fit in memory and everything downstream gets better.
Here's what we actually found: a 6-billion-parameter flash model beat a 32-billion-parameter
coder-tuned model on the same coding task — when each was paired with the right engine.
Model quality, as most people measure it, wasn't the variable that mattered.
The lab
At QCoda we run structured experiments before we ship anything — our version of a structured lab. Each lab has
a hypothesis, a test plan, and a graduation criterion. LAB-009 asked a specific question: when you
pair different agentic coding engines with different local models, what actually determines whether
the task succeeds?
We tested five engines: aider (in two modes — whole-file and diff), goose, opencode, pi, and
claude-code. Seven local models: qwen2.5-coder:32b, qwen3-coder:30b, gpt-oss:20b,
devstral-small-2:24b, glm-4.7-flash, gemma4:26b, gemma4:12b. Ten trials per engine-model pair.
Same task each time: add four specific routes to an existing Flask application. Same context — the
real spec files and project ledger passed in, the way QCoda actually runs in production.
Pass means all four routes present, file parses, no existing code dropped. Fail means anything
else. The pass-rate is what it is.
Here's the matrix:
| Model |
aider:diff |
aider:whole |
goose |
claude-code |
pi |
| qwen2.5-coder:32b |
10/10 (21s) |
10/10 |
0/10 |
0/10 |
5/10 |
| qwen3-coder:30b |
10/10 |
0/10 |
10/10 |
10/10 |
9/10 |
| gpt-oss:20b |
10/10 (59s) |
9/10 |
5/10 |
9/10 |
9/10 |
| glm-4.7-flash |
8/10 (474s) |
6/10 |
7/10 |
8/10 |
10/10 (24s) |
| devstral-small-2:24b |
10/10 |
— |
1/10 |
— |
10/10 |
The columns don't tell a consistent story by model. The rows don't tell a consistent story by
engine. The interactions are the finding.
The edit-format revelation
The biggest result in the matrix isn't about which engine is best. It's about why aider's two
modes produced opposite outcomes on the same model.
aider:whole went 0/10 on qwen3-coder:30b. aider:diff went 10/10 on the same model. Same model,
same context, same task. The only difference was the format aider used to request the edit.
Whole-file mode works like this: aider hands the model the existing file and says "here's what's
in it, give me back the complete new version." Under real context load — actual spec files, actual
project ledger, the stuff QCoda feeds into every production run — a 30B model can't hold all of
that and re-emit a 195-line file. It drops the task. It applies zero edits. It exits
cleanly. No error.
That's not a model bug. That's asking too much of the format.
Diff mode (SEARCH/REPLACE) works differently: aider says "find this exact block, replace it with
this." The model emits only the changed hunk. It never re-emits the file body — so it physically
cannot drop existing code. Under any context load, on every model we tested, aider:diff produced
zero truncation events across 18 cross-complexity runs.
Zero.
That's not a statistical result. That's a structural property of the format.
Here's the part that matters: the "model is dropping my code" bug — the one where a file goes
from 195 lines to 10 and the agent reports success — is overwhelmingly an edit-format problem, not
a model problem. We saw this problem before on projects. We called it the gutting failure. We blamed models.
We added guardrails. The actual fix was changing one flag:
--edit-format diff.
The GLM surprise
Look at the glm-4.7-flash row in the matrix. It's a flash-class model — fast, small, not
coder-tuned in the way qwen2.5-coder is. If you're selecting models by benchmark score or parameter
count, glm doesn't make your shortlist for a serious coding task.
Pi went 10/10 on it at 24 seconds.
That's the fastest clean result in the whole matrix, on a model you'd expect to struggle.
Meanwhile, pi went 5/10 on qwen2.5-coder:32b — 32 billion coder-tuned parameters — with two
truncation events.
The reason connects directly back to Episode 1. Pi routes through a structured tool-call
interface. glm-4.7-flash emits structured tool_calls natively — no normalization
needed, clean protocol compliance out of the box. qwen2.5-coder:32b, behind raw Ollama, emits
JSON-in-content. Pi parses the interface, finds no structured call, and works with degraded
information. Sometimes that's fine. Half the time it isn't.
That's not a size story. That's not a benchmark story. It's a protocol compliance story.
The model that wins on pi isn't the smartest model. It's the model that speaks pi's language.
Heavy loops vs. low round-trips
Claude-code was the most interesting engine to measure, because it revised its own prior verdict
during the lab.
The initial runs against devstral-small-2:24b with faithful context were not encouraging: 0/5. We had
logged a conclusion — "claude-code not viable on local OSS for real tasks." Then we ran it against
qwen3-coder:30b: 10/10. Against gpt-oss:20b: 9/10.
The prior "not viable" conclusion was an artifact of testing it only on models that couldn't
support its loop. Claude-code's multi-round agentic pattern — Read → Plan → Edit → Verify — burns
through a lot of turns. On a capable 30B model, those turns complete and the edits land. On a
weaker or protocol-incompatible model, the loop spends its budget reading and planning, and the
headless session ends before any edit is applied.
That's the heavy-loop problem. And it affects opencode too, which timed out at 1200 seconds on
qwen3-coder:30b — not because opencode is broken, but because its read-edit-verify-lint cycle on a
local 30B model running on commodity hardware runs out of runway before it finishes. (Opencode's
0/10 elsewhere is an unresolved invocation issue, not a discipline result — we're not counting that
against it.)
On local hardware, round-trip count is a first-class constraint. Low-round-trip engines —
aider:diff, pi — win on practicality, not because they're more sophisticated. Aider:diff finishes
in 21 seconds on qwen2.5-coder:32b. That's not just faster. It's 57 times cheaper in GPU
wall-clock than a 1200-second timeout, for a better outcome.
What this means in practice
The question "which model is best for coding?" is underspecified to the point of uselessness.
Better question: which engine are you running? Does your model emit structured tool calls
natively? What's your context load? How many round-trips does your hardware support in a reasonable
timeout?
These are infrastructure questions, not model questions. And if you're choosing models without
having answered them, you're debugging the wrong layer.
At QCoda, the lab results directly fed a routing function — _resolve_edit_format()
— that picks the right aider mode per model capability automatically. The operator doesn't configure
this. The orchestration layer knows from the capability index which models get diff and
which get whole. It's one of the first expressions of a broader bet: the right tool
for the right task is an orchestration decision, not a manual one.
That bet gets more interesting when you add task complexity into the routing signal. Which is
Episode 3.
QCoda Labs runs structured experiments to understand what actually works in local
AI-assisted software development. This series documents what we found in LAB-009 — engine and model
compatibility at scale. All runs: local Ollama endpoints, no cloud inference, N=10 per cell.
