The Best Agent Scores 25.6 Percent

An earlier post here covered what an eval is in general terms. The general terms have been overtaken. Three things happened this summer that change how you should read any agent score you are shown, including your own.

1. The leader passes a quarter of the tasks

Harbor-Index shipped on 29 June 2026. It distils 82 tasks from 6,627 candidates across 54 benchmarks, selected to be lightweight, diverse and difficult, and it scores pass rate against reconstructed cost per run on a log scale.

The top result:

| agent and model | pass rate | | --- | --- | | GPT-5.5 on the Codex CLI | 25.6% | | Opus 4.8 | about 9% | | Gemini 3.1 | about 7.3% | | Open-weight models (GLM, Kimi, MiniMax, Qwen, MiMo, DeepSeek) | 7% to 12% |

The best combination anyone tested fails three quarters of the tasks. The rest fail nine out of ten.

Sit with the gap between that and the marketing you have been reading. These are not adversarial trick questions; they are tasks distilled from benchmarks the field already agreed were reasonable. The honest summary of the state of agentic systems in late 2026 is that they are genuinely useful and they fail most of the time, and both halves of that sentence are load-bearing.

Senior SWE-Bench, 16 July 2026, points the same way from a different angle. It grades code *quality* on under-specified real pull requests merged after February 2026, not just whether tests pass. The leader, Claude Fable 5, scores 29.1%. Even at the top, more than seventy percent of tasks fail.

The practical read: if your agent pipeline has no human review step and no machine-checkable gate, you are shipping the failure rate straight into production. The published numbers do not support any other conclusion.

2. Benchmarks became software, and that is good news

Terminal-Bench shipped 3.0 on 30 July 2026, Terminal-Bench-Science 0.1 on 27 August, and 4.0 on 28 August. The 4.0 release notes are the interesting part: a flat eight-hour timeout, eight broken tasks removed, nineteen fixed.

Their stated principle is worth quoting: benchmarks are software and should be maintained like software.

This is a real shift and it has a consequence people keep getting wrong. A benchmark score is only meaningful with a version attached. "We score X on Terminal-Bench" is now as underspecified as "we run Postgres." Comparing a number from 3.0 against one from 4.0 compares two different task sets, one of which had eight broken tasks in it.

If you publish agent scores, publish the benchmark version. If you read them without one, treat them as approximate.

3. The way you validate an LLM judge was probably wrong

This is the finding with the widest blast radius, because most teams evaluating agents use a model as the grader.

A paper published 17 June 2026 (arXiv 2606.19544) ran 21 judge models across roughly 541,000 judgments. It compared the usual validation method, exact-match agreement with human labels, against Cohen's kappa, which corrects for agreement that would happen by chance.

The gap on MT-Bench was 33 to 41 percentage points.

That is not a rounding error. A judge reported at "eighty percent agreement with humans" can be barely above chance once you account for the base rates. The standard validation method systematically overstates judges, and it does so by enough to invert conclusions.

The paper also documents a consistency and bias paradox: judges with strong test-retest reliability that simultaneously show strong position bias. A judge can be reliably wrong in the same direction every time, and a reliability check will pass it.

A second paper, 26 August 2026 (arXiv 2608.25869), found anchoring: showing a judge a prior score contaminates its own. If your pipeline shows the grader the previous run's rating, or grades in a rolling context that carries earlier scores, your numbers are correlated by construction.

Three things follow, and they are cheap to do:

• Report kappa, not raw agreement. If you only have raw agreement, you do not know what you have.
• Randomise position in every pairwise comparison, and check whether the result moves when you swap the order. If it does, the position bias is yours now.
• Grade each item in a clean context. No prior scores visible, no rolling history.

What people are doing instead

Two responses shipped, and both move away from grading one final outcome.

Behavior specs, published as an open standard by Braintrust and Basis on 29 July 2026 at agentbehavior.dev, grade hundreds of intermediate decisions along a trajectory rather than one hard-to-verify end state. The reasoning is straightforward: an agent's final answer is often difficult to grade, while "did it call the right tool with the right arguments at step 12" is not.

This is process supervision, and it fits agents far better than outcome scoring does. It also produces something you can debug. A failed trajectory tells you where it went wrong; a failed outcome tells you only that it did.

smevals, Simon Willison's YAML-defined eval CLI, shipped 31 July 2026. Its stated motivation is the useful one: finding the cheapest model that is good enough for a given job. That framing pairs directly with Harbor-Index plotting cost against pass rate, and with the cost per finished task argument.

How to read any agent number from here

A short checklist, all of it earned from the above:

1. What version of the benchmark? Without it the number is approximate.
2. What did the run cost? A score without a cost is half a result.
3. If a model graded it, what is the kappa? Raw agreement is not evidence.
4. Was position randomised, and was the grading context clean? If not, expect bias in a known direction.
5. Outcome or trajectory? Outcome scores on agents hide where the failure was.
6. What is the pass rate honestly? If someone reports a number far above 25.6% on genuinely hard, diverse tasks, the first question is what made their task set easier, not whether their agent is better.

None of this says agents do not work. Bun rewrote a runtime with them. It says the measurement layer was weaker than the thing it was measuring, and that gap is now documented well enough that you cannot claim not to know.