SWE-bench Verified is the benchmark most people quote when they say a coding agent is good. It is 500 human-reviewed GitHub issues from popular Python repositories, scored by actually running tests, which makes it far better than an opinion poll. It also saturates, disagrees with itself across leaderboard sites, and does not survive contact with your repository.
What the benchmarks actually measure
| Benchmark | What it is | Size | The catch |
|---|---|---|---|
| SWE-bench Verified | Real GitHub issues from Python repos, scored by running tests | 500 instances | Top scores cluster so tightly the differences are noise |
| SWE-bench Pro | Harder, built to resist contamination | Larger | The best results sit in the mid 50s rather than the 80s |
| Terminal-Bench 2.0 / 2.1 | Terminal tasks with a human-written solution and a unique environment | 89 tasks | A refresh fixed environment gaps, so older and newer scores are not comparable |
| Terminal-Bench 4.0 | Agent plus model, run for cost as well as completion | 330 trials each | Measures the pair, not the model alone |
Four numbers that all get called the coding benchmark score in casual conversation.
The first thing to notice is that a model-level score and an agent-level score are different quantities. On the same 2.1 tasks, one reporting site has a model at 91% on a fixed harness while the agent board for the same tasks has a pairing at 87%. Both are correct. They are answers to different questions.
Image goes here
/blog/how-to-read-coding-agent-benchmarks-scaffold.png
Same model, three different harnesses, three different scores
Show generation prompt
Editorial data illustration, 16:9, flat vector. One large circle at the top (a single model) with three arrows fanning down to three progressively larger trapezoids (scaffolds of increasing size), each ending in a bar of a different length representing a different score. Charcoal (#121212) background, grey (#8c8c8c) outlines, the longest bar in indigo (#5560d4), ordered-dither dot texture, generous negative space, centered composition. Mood: comparative, revelatory. No readable text, no logos, no watermark.
Why the scores disagree
Check the same benchmark across three leaderboard aggregators and you can find leading figures ten points apart for what is nominally the same measurement. That is not anyone lying. It comes from different scaffold versions, different reasoning budgets, whether thinking blocks were included, whether the number is self-reported by the vendor or independently measured, and periodic changes to the benchmark harness itself.
Some benchmark teams have responded by marking a benchmark archived once it saturates, on the grounds that a metric where everything scores in a narrow band at the top no longer discriminates. When the spread between models is smaller than the spread between measurement methods, ranking products on it is guesswork with decimals.
The transfer problem
Every item in these benchmarks is a Python repository with an issue and a failing test, and the agent gets to start from a clean state with a well-defined success condition. Your work has none of that. It has a monorepo, a flaky suite, an undocumented build step, a colleague's half-finished refactor and a service that only starts on port 3000 on Wednesdays.
The guidance that has crystallised from this is blunt: a 70% Verified score does not survive contact with your repo. What survives is the method that replaces it. Take your own last 50 merged pull requests, reset to each parent commit, give the agent the original issue, run it against a budget, and compare the result to the patch that actually shipped.
- Exclude version bumps, generated files and lockfile churn.
- Record the parent commit, the issue text, the merged diff and the test delta.
- Run each configuration in an isolated worktree with a fixed step and cost budget.
- Score behavioural equivalence: would this patch have passed the same review?
- Compare score against cost and steps, then refresh weekly.
What to measure instead
Three numbers beat a public leaderboard for your own decision. How many of your tasks reach a state a reviewer accepts without a rewrite. What each accepted change costs in tokens and in wall clock time. And how much of the diff you end up keeping, because an agent that writes 600 lines to fix 40 is cheap on tokens and expensive on review.
That last metric connects to the real bottleneck. Review time, not generation time, is what teams report slowed down in 2026, which is the subject of the verified completion post.
Where Quaka fits
Quaka does not publish a SWE-bench score, because Quaka is a harness and any score would be a score for whichever model you picked behind it. What it does own is the parts of the measurement that are harness-level: whether the run's tests were really executed, whether the diff matches the claim, what the run cost, and how many steps it took.
Because threads are provider-neutral, you can run the same prompt on several models from the same UI and compare the outcomes side by side against your own repository rather than someone else's Python issues.
Get an invite
Quaka is in private beta.
One email when your invite is ready. macOS, Windows and Linux, bring your own key.
Read the source on GitHubCommon questions
Is SWE-bench Verified a reliable way to choose a coding model?
It is a reasonable signal for raw capability and a poor signal for your use case. It uses Python repositories, clean starting states and clearly defined success conditions, and it has saturated to the point where several leaderboard sites no longer run it on new models. Treat it as a floor, not a ranking of what will work in your codebase.
Why do different sites report different scores for the same model?
Because the harness, reasoning budget and evaluation version differ, and because some figures are self-reported while others are measured independently. Scaffold upgrades alone have been shown to move scores significantly, which is why a 100-line agent scoring 65% on Verified is entirely consistent with a frontier model scoring far higher in a heavier scaffold.
How do I benchmark a coding agent on my own code?
Replay your own recent history. Take the last 50 merged pull requests excluding version bumps, reset to each parent commit, give the agent the original issue, run it in an isolated worktree with a step and cost budget, and compare its patch against the one that shipped. Score it on whether a reviewer would have accepted it, then refresh weekly.