How we evaluate coding models

We serve a lot of coding traffic, and every few weeks a new model shows up claiming to be better at it. Deciding whether to switch is not a research question — it is an operational one, and it recurs.

The tempting way to decide is a price table. Prices are published, exact, and comparable to the cent. But price answers a different question than the one we have. What we need to know is whether a model will do the work well, and the only honest way to find that out is to make it do the work and look at what comes back.

So we built an eval suite. This post is about what it measures, and — more usefully — about three times it confidently told us something false before we caught it.

What the suite is

108 tasks, run against the live proxy rather than against a provider's API directly, so a score includes everything a real request goes through: our system prompt, our request handling, our error paths. If the stack degrades the model, we want that in the number.

The tasks split into two kinds that are scored very differently.

21 execution tasks. The model is given a bug or a feature request; its response is parsed for code, that code is run against reference pytest tests in a subprocess, and the score is the fraction of tests that passed. No opinion involved — the tests pass or they do not.

87 judge tasks. Behavioural things you cannot unit-test: did it plan before editing, did it verify its claim, did it stay in the language it was addressed in, did it do the small thing it was asked for instead of the big thing it imagined. These are scored by another model against a rubric.

The two combine as composite = 0.7 × exec + 0.3 × judge. The weighting is a judgement call: execution is the harder signal to argue with, so it carries more, but a model that writes passing code while ignoring every instruction is not one we want.

One tradeoff worth stating plainly: the reference tests run in a plain subprocess with a stripped environment and a timeout, not in Docker. We are only ever executing code our own models produced for our own fixed prompts, so the threat model is thin. The day the tasks ingest untrusted text, that has to change.

The shape most coding evals get wrong

Every task in this suite is written from scratch. That is the easy half. The harder half is getting the shape right, and for a long time we did not.

A typical eval task looks like a question: a human describes a problem, the model answers. Almost no turn in a real coding agent looks like that. In an agent loop the model is usually picking up after a tool returned something — a file it just read, a test that just failed, a command's output — and the decision being scored is what to do next. The history is long, native tool definitions are attached, and most of the prompt is context the model did not write.

Our early suites were almost entirely question-and-answer. That is a real shape, but it is the minority one, and a model can be good at it while being poor at the loop. Fourteen tasks now open on a tool result with no trailing human turn at all, which meant teaching the harness to send native tool histories and to score a response where the prompt is empty by design.

It is still the limitation we would attack first. A suite of one-shot tasks also cannot see conversation-level effects — how a model behaves on turn 30 of a session is not something 108 independent requests will ever tell you.

Three ways we fooled ourselves

1. A token ceiling, scored as quality

One of the models we run was retrained in place — same model id, same API, new weights, announced in a changelog. We re-ran the suite to see what changed, and the category scores looked alarming. Devops fell from 0.694 to 0.000. Code review halved. Planning dropped by a third.

The story wrote itself: the retrain traded instruction-following for benchmark scores. It was a good story and it was wrong.

The retrained model reasons considerably more, and reasoning tokens count against the same max_tokens budget as the answer. On 12 of 108 tasks it spent the entire 8,000-token allowance inside the reasoning block and returned empty content. Our client did not read finish_reason, so an empty string went to the judge, which duly scored zero on every criterion — and a zero from the judge is indistinguishable, downstream, from a zero the model earned.

Devops has two tasks in it. One truncated, one timed out. Hence 0.000.

The fix has three parts. An empty response with finish_reason: length is now recorded as a failed measurement and excluded, not scored zero — because "the model answered badly" and "the model did not answer" are different facts and must never be averaged together. An empty response with finish_reason: stop still scores zero, because that is the model choosing to say nothing, which is a real quality signal. And the budget went to 32,000, which is the order real clients send anyway.

There is a second lesson in that episode which has nothing to do with evals. Pinning a model id does not pin its behaviour. The same string, at the same endpoint, started returning a model that reasoned several times harder and emitted roughly seven times the output tokens for identical work — better at the job, and a different thing to budget for. Nothing in the request had changed. If your product depends on a specific model, that is a change somebody has to notice, and the noticing has to be systematic, because the announcement will not always arrive before the weights do.

2. A composite that did not say what it rested on

The deeper problem was that nothing in the artifact recorded how much of the suite had actually been measured. A task that fails is excluded from the mean, so a run can post a perfectly healthy composite while a seventh of it silently did not happen.

Re-auditing our existing runs with that check, three of six were not usable. Two had lost tasks to a 120-second client timeout — infrastructure wearing a score's clothing.

3. A threshold that was not a test

Our comparison tool called a result significant if the composite moved more than 0.06. That number came from eyeballing run-to-run variance once, and it had been quietly deciding things ever since.

When we re-measured the retrained model properly, it improved by 0.040. The tool called that "within noise" and moved on.

But both runs answered the same 108 tasks, which means we can do much better than comparing two independent error bars. Resampling the per-task differences cancels the variance between tasks — a hard task is hard on both sides, and contributes nothing to the uncertainty of the difference. The paired interval came out at +0.007 to +0.084, never touching zero, with the improvement holding in 99.6% of resamples.

A real improvement had been sitting behind a fixed threshold that had no idea how big the suite was or how large the effect might be. The threshold is now a fallback for cases too small to bootstrap, and the verdict records which rule produced it.

What the ranking actually says

With the harness fixed, we ran seven models through the same 108 tasks with the same judge. Here is the result, with 95% intervals:

That is the honest headline, and it is a negative result: our suite cannot order these models. It can eliminate one. Ranking the other six by their point estimates would be reporting precision we do not have.

Two further checks made that conclusion harder to escape.

The first is that the judge was itself one of the contestants, and it came first. So we re-scored the top three with a neutral judge that was not in the ranking. Kimi K3 stayed first — its lead is not self-flattery. But second and third swapped. An ordering that changes when you change who is marking is not a property of the models.

The second is resolution. Half our categories hold four tasks or fewer, where a single task flipping moves the score by 0.25 or more. Devops, the category that started this whole investigation, has two.

What the average was hiding

A single number per model is a convenient thing to rank and a poor thing to learn from. Broken out by stage, the models stop looking like a league table and start looking like different instruments.

The columns are not flat, and they are not the same shape. Among the categories with enough tasks to trust: one model is clearly the weakest at holding the language it was addressed in — a gap of more than 0.2 against the best, on nine tasks. A different model is the weakest at respecting the role its harness assigned it, which is the largest category in the suite. Two others fall off sharply on restraint — doing the small thing asked rather than the large thing imagined — while scoring perfectly well elsewhere.

This is why the composite cannot separate them. Each model is paying for its strengths with a different weakness, and a weighted mean is exactly the operation that cancels that out. If you are choosing a model for one specific job, the row that matters to you is probably not the row the average is made of.

The axis that does separate

Quality did not separate these models. Cost separates them by two orders of magnitude.

Kimi K3 costs roughly 100× what DeepSeek V4 Flash costs, for a quality difference this suite cannot demonstrate exists. That is not an argument that K3 is a bad model — it may well be better in ways 108 tasks cannot see. It is an argument that if you are going to pay 100×, you should be able to say what you are buying, and we cannot.

It also means the shape differences above are the actionable output of this whole exercise, in a way that "0.9608 versus 0.9529" never was.

If you are building one of these

Four things we would do from the start next time.

Record coverage next to every score. A composite with no statement of what it rests on is not a measurement. Ours now carries scored, excluded, truncated and timed-out counts, and a run below 97% coverage is refused a place in the ranking rather than quietly included.

Separate "answered badly" from "did not answer". They arrive looking identical — an empty string and a low score — and averaging them together is how a ceiling becomes a quality claim.

Pair your comparisons. If both runs did the same tasks, resample the differences, not the scores. It is a few lines of code and it turned a "no result" into a result for us.

Never let the judge be a contestant without checking. Run the top of your table past a judge with nothing at stake. If the order moves, you have learned something important about your table.

And the meta-lesson, which is the one that keeps recurring: every time this suite has been wrong, it has been wrong in the direction of a clean story. Devops collapsing to zero was a better story than a token ceiling. A fixed threshold saying "no change" was a tidier outcome than a confidence interval. When your eval hands you a headline, that is the moment to go and check the plumbing.

Why we bother

Because we are the ones who have to answer the question, repeatedly, for other people's work.

Read the results back and the useful part is not the ordering — it is that the ordering barely exists. Six of seven models are statistically indistinguishable on this suite while their prices span two orders of magnitude. Model choice, for most coding work, is not the high-stakes decision it is usually presented as. What actually moves is quieter: a provider retrains under the same id, output volume jumps, a model that was strong at holding its instructions turns out to be the weak one at holding its language. None of that shows up in a launch post, and none of it is visible from inside a single project.

Watching for it is work — a suite, a judge, a statistics pass, and the discipline to re-run all of it when something moves. It is worth doing once for a lot of people and hard to justify doing for yourself. That is the whole proposition of Claudinio: you point your existing tools at one endpoint, we keep measuring what sits behind it, and you get on with the code. If you would rather run your own suite, everything above is how we would start.