Evidence note: this article rests on two clearly different grades of material, and the distinction holds for every section below. The vendor and project announcements — a hardware vendor’s competition write-up, a Google developer blog post, a GitHub changelog entry, a public project’s own tutorial, and two encyclopedia entries on historical benchmarks — are primary: the fact being reported is that the party in question published the statement. The December 20, 2025 model evaluation at the centre of this piece is a different grade entirely. It comes from a single community post and has not been independently reproduced or corroborated, so every figure drawn from it is marked at the point it appears. No experiment was run for this article.
What reportedly ran on December 20, 2025
A benchmark described as TRIATHLON-LLM is reported to consist of 50 questions spanning 10 cognitive dimensions, with a test run dated December 20, 2025 (single community post; not independently corroborated). The scale on which results are expressed is 125 points rather than 50, which implies roughly 2.5 points per question — a derived reading of the published numbers, not a weighting the author is reported to have explained. Sources do not confirm how partial credit was assigned, who or what graded the answers, or whether any question was run more than once per model.
That matters before any score is discussed. A 50-item instrument has coarse resolution by construction: on a 125-point scale, a single fully-credited question is worth about 2.5 points (derived), which is already most of the reported top-to-bottom gap. Readers comparing frontier models on this basis are working with an evaluation whose smallest unit of measurement is close to the size of the effect it found.
The ten dimensions, named
The ten reported dimensions are logic puzzles with deliberately tricky wording; mathematics covering Bayesian reasoning and combinatorics; code debugging and system design; science explanations under stated constraints; causal reasoning; language nuance; creativity under constraints; applied ethics; hallucination traps; and adversarial prompts (same single source; the category list has not been independently verified).
Read as a list of intentions rather than a validated taxonomy, it is a reasonable sketch of what a working developer actually leans on — and notably, three of the ten (debugging and system design, hallucination traps, adversarial prompts) are failure-mode categories rather than capability categories. Whether 50 items can carry ten dimensions at a usable per-dimension sample size is an open question the available material does not answer; at an even split that is five questions per dimension (derived).
Why the author is said to have built it at all
The stated motivation, as reported, was dissatisfaction with synthetic benchmarks such as MMLU and HumanEval — instruments the author is described as saying measure something, but not what matters in day-to-day use of a language model (author’s stated reasoning, single source, uncorroborated).
This is the most durable part of the artifact, and it survives whatever happens to the scores. The complaint is about construct validity: not that the established benchmarks are wrong, but that what they declare as their subject has drifted away from what a developer needs to predict. That question has been asked from other directions too — see this site’s earlier look at what LLM benchmarks are actually measuring — and it is not settled by anything in this cycle.
123, 120, 120 — and the three points in between
The reported results: Gemini 3 at 123 of 125, or 98.4%; Claude Opus 4.5 and Grok 4.1 each at 120 of 125, or 96.0% (single-source scores from the December 20, 2025 run; not independently reproduced). The top-to-bottom spread is reported as 3 points, or 2.4% — an arithmetic relation that does check out against the stated scores (3 ÷ 125 = 2.4%, derived).
Three points. On an instrument whose question granularity appears to be about 2.5 points, a 3-point gap is approximately one question plus a fraction. Without repeated runs, a reported confidence interval, or a second grader, the available material does not establish that the ordering would survive a re-run — and sources do not confirm that any re-run was attempted. The honest reading of a 2.4% spread from n=50, single pass, is that it is consistent with a real small difference and equally consistent with grading noise.
Where nothing moved
All three models are reported to have scored identically in 7 of the 10 categories, with mathematics and code among the ties, and all three are reported to have refused to hallucinate and resisted every adversarial prompt (single source, December 20, 2025; uncorroborated).
If that holds, the two categories a developer most often shops on — mathematics and code — are reported to be exactly where the instrument found no difference at all. A clean sweep on hallucination traps and adversarial prompts is also the signature of a ceiling: when every subject passes every item, the items have stopped discriminating, and the result describes the test rather than the models. That is not a criticism of the author’s design so much as the predictable fate of any fixed small set pointed at a fast-moving frontier.
What a benchmark used to name as its subject
Two historical instruments are worth setting beside this one, because both were explicit about their subject in a way modern evaluations rarely are. The LINPACK benchmarks are described as measuring a system’s floating point architecture — a named, physical property of the machine (encyclopedia entry; treat as a summary rather than a primary specification). The Rugg/Feldman benchmarks are described as consisting of seven short programs written in BASIC (same caveat).
Seven short programs is a smaller instrument than fifty questions, and nobody mistook it for a measure of general machine intelligence. Its subject was stated and narrow, and its score was interpretable because of that narrowness. The December 20, 2025 table inherits the small-n shape without inheriting the narrow declared subject: ten cognitive dimensions is a very large claim to hang on fifty items.
Where a difference was still measurable this cycle
Two places, and neither of them is the model.
NVIDIA’s KGMON team is reported to have taken second place in the KDD Cup 2026 Data Agents competition with a system built around making the agent’s harness smaller (NVIDIA’s own engineering blog; the competition placement is the vendor’s account and has not been independently checked here). The lever there was scaffolding, not model selection.
Separately, Google’s developer blog reports an unattended overnight run of Gemini 4 Argon against a large Python codebase using Google’s open-source Agent Development Kit, cutting in-memory session preparation overhead by 40% in long sessions and speeding cold imports by 21% (vendor-reported figures from the vendor’s own blog; no independent reproduction). Those are two-digit improvements in a cycle where the model comparison produced 2.4%.
The pattern is worth stating plainly: the measurable wins this cycle came from changing the harness and from letting an agent work on a codebase unattended — both things a small team controls — while the model choice produced a difference inside the noise floor. PlayAgit’s earlier coverage of a harness that shrank its way to second place traces the same seam.
A method arrives with no leaderboard attached
Running alongside all of this is a verification route that produces no score. Formal specification languages such as TLA+ are discussed as a way to check AI-generated code against real production data through trace validation and model-based testing (reported in an engineering-blog argument; the applicability to agent-written code is argued there, not demonstrated under controlled conditions).
Trace validation inverts the comparison. Instead of asking which model is better in general, it asks whether this specific system’s observed behaviour conforms to a specification someone wrote down — a question that has an answer even when every candidate model scores 96%. It is also the one approach in this set that gets stronger as models converge: when the generator is interchangeable, the check is what remains. The habit of agent systems naming their own checking step shows up repeatedly, as this site’s survey of three such systems found.
What that does to "choose on price and ecosystem"
The author’s reported conclusion is that model choice should rest on price and ecosystem rather than benchmark scores, because frontier models have become statistically indistinguishable (author’s stated conclusion, single source, uncorroborated).
Half of that advice is perishable, and a date shows why. GitHub’s changelog states that a selected set of GitHub Copilot models will be deprecated across all Copilot experiences, with the removal dated November 16, 2026 (platform changelog entry; the specific date and model set have not been independently confirmed, and earlier dates have circulated — PlayAgit’s own prior coverage cited October 19, 2026). Readers tracking that schedule can compare the earlier account in this site’s report on the Copilot retirements; the discrepancy between the two dates is itself the point.
"Ecosystem" sounds like a stable property. A dated removal across every surface of one vendor’s assistant product is evidence that it is not. Choosing on ecosystem means choosing something a vendor can revise on a published schedule, and the revision does not wait for a benchmark cycle.
The question a cross-provider code benchmark is actually asking
A separate project called llm-benchmark documents itself as benchmarking LLM-optimized code across multiple providers; that framing comes from the project’s own tutorial, and the fact being reported here is what the project states about itself, not an independent assessment of its results.
Read against a November 2026 removal date, a cross-provider code benchmark is not really asking which provider wins. It is asking how much work it is to leave — whether the same prompts, the same tool definitions, and the same expected outputs survive a provider swap. That is a portability measurement dressed as a quality measurement, and it is the more useful of the two when the quality gap is 2.4%.
A follow-up proposed here, not reported
No experiment was conducted for this article, and the following is a proposal only. The obvious next step is to apply trace validation to agent-written code produced in this project’s own workflow: write a specification for a small, well-bounded component, collect execution traces from real runs rather than synthetic inputs, and check conformance — then repeat the identical procedure with a second provider’s model as the generator. What that would establish is whether provider substitution changes the conformance rate at all, which is the question the score table cannot reach. PlayAgit has a usable corpus for it — the agent-run record behind three Roblox games built across 303 runs — but nothing has been measured, and no result should be inferred from this paragraph.
Developer takeaway: fix what cannot be withdrawn on a schedule
For a solo developer or small team picking one frontier model on a real budget, the material in this cycle points in a consistent direction.
The reported 2.4% spread, if it holds, is too small to carry a procurement decision — and it was reportedly zero in mathematics and code, the two categories most often cited to justify one. Price, rate limits, and context window are the differences large enough to measure from outside, and they are also the ones a published invoice will confirm.
What is worth investing in is the part with your name on it. The two two-digit improvements reported this cycle both came from the harness and the run conditions, not from the model: a competition placement attributed to shrinking the agent’s scaffolding, and an overnight unattended run reporting 40% less session-preparation overhead and 21% faster cold imports (both vendor-reported, both uncorroborated here). A verification layer — specifications, trace validation against real production data, conformance checks that do not care which model generated the code — is the asset that survives a vendor’s lineup change, and November 16, 2026 is a reminder that lineup changes arrive with dates attached.
Put concretely: treat the model as a replaceable component and write down, in testable form, what its output must satisfy. Then a model retirement is a configuration change rather than a rewrite — and a 3-point difference on a 125-point scale becomes what it probably already is, which is not a decision.