🔍 Read the full analysis: FrontierCode: What Your AI Needs To Get A Pull Request Reviewed on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 AI models on whether repository changes would meet maintainer review standards. Claude Opus 5.5 led with a 52.2% overall score, but cleared all blocking checks in 57.7% of attempts; the benchmark also found large cost differences between models.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark that measures whether coding agents produce changes a maintainer could accept, rather than simply whether code passes tests. Claude Opus 5.5 scored highest at 52.2%, but it cleared every blocking check in only 57.7% of attempts, leaving substantial room for human review.
FrontierCode evaluates code changes against checks for hidden-test correctness, regressions and code cleanliness, whether the agent’s tests catch the original bug, adaptive tests for complex behavior, limits on files and diff size, and an LLM-based review against maintainer criteria. Failing any blocking check gives an attempt a zero; passing all blocking checks earns a score from 70% to 100%, adjusted for quality. The benchmark also penalizes oversized changes, edits to unrelated files and unhelpful tests.
In Artificial Analysis’s results, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%, followed by GPT-6 Astra at 47.0%. Xiaomi’s MiMo-V2.6-Pro led the open-weight models at 45.4%, with DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. Nine models from six labs scored between 43% and 52%; scores then fell, with Grok 4.7 at 33.8% and Nemotron 3 Ultra at 10.5%.
The evaluation used the same open-source agent harness, Stirrup, for each model, with up to 250 turns per attempt and three attempts per task. The review component used GPT-6 Luna (medium) as its judge. Artificial Analysis ran the benchmark’s 100-task Main set; the full FrontierCode benchmark contains 150 tasks.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
A Passing Test Is Not a Merge
The result points to a gap between producing code that appears to work and producing a patch that meets a repository’s review standards. The leading model failed at least one blocking check in 42.3% of attempts, based on the reported 57.7% pass rate. That makes the benchmark relevant to teams considering whether coding agents can submit changes with little or no human oversight: these results do not support treating a high leaderboard score as proof that review can be skipped.
FrontierCode also indicates that model choice depends on the task being measured. In the source material’s comparison with Artificial Analysis’s Intelligence Index, Claude Sonnet 5.5 ranked second among the listed models on the general index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. Conversely, DeepSeek V4.1 Flash performed better in this coding evaluation than its general index position might suggest. General capability rankings alone may not predict review-ready code performance.
Cost is another practical consideration. Artificial Analysis reported per-task costs from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. On its cost frontier, it listed GPT-6 Luna at 41.9% for $0.09, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus at 52.2% for $10.15. The 3.1-point score difference between Sol and Opus came with a much higher reported cost; results do not establish which model is best for any particular team or codebase.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July, adding rules against agents looking up task solutions online. According to the source material, more than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The tasks are private, reducing the chance that models encountered them during training.
The benchmark’s design attempts to test more than whether a patch passes ordinary tests. Reverse tests, for example, check that an agent’s own tests fail against the unmodified code; otherwise, a test that passes both before and after a change may not demonstrate that the reported problem was fixed. Scope limits and the review check are intended to capture concerns maintainers face when evaluating changes. Cognition’s trajectory scanner can also assign a zero if it flags an attempt for seeking the solution.
Artificial Analysis’s results cover its 100-task Main set, not all 150 FrontierCode tasks. The source material says nine of the 100 tasks were unsolved by every model on every attempt. That result reflects performance under this benchmark’s setup; it is not a measure of every coding task or repository.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s FrontierCode announcement
As an affiliate, we earn on qualifying purchases.
Models and Tasks Still Missing
The results do not include every prominent model. As of October 13, the source material lists Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result. Their performance cannot be inferred from other coding evaluations or general intelligence rankings.
It is also unclear how closely scores from the 100-task Main set predict performance across the full 150-task benchmark, or how they would translate to a specific company’s repositories, review policies and agent setup. The source describes the per-task costs and evaluation configuration, but does not establish that those costs or rankings will hold for other workloads. Benchmark scores are evidence about this test setup, not guarantees of production outcomes.
As an affiliate, we earn on qualifying purchases.
Awaiting More Benchmark Results
The immediate next development to watch is whether Artificial Analysis adds results for the models that were missing as of October 13, including Muse Wonder Full and Gemini 4 Argon. Further model results could change the rankings and cost comparisons. No release date for those evaluations is specified in the available source material.
For now, the published scores offer teams a comparative test of coding agents under a shared harness, while leaving repository-specific evaluation to users. The current findings support checking patches for correctness, scope and review quality rather than relying on a model’s overall ranking or test-pass claims alone.
software development code analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
FrontierCode measures whether an AI-generated repository change meets a set of maintainer-style review checks. These include hidden tests, regression and cleanliness checks, test quality, change scope and an LLM-based review.
Which model scored highest in the reported results?
Claude Opus 5.5 (max with fallback) led with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts, according to Artificial Analysis’s results.
Does a high FrontierCode score mean an AI can merge code without review?
No. The leading model cleared all blocking checks in 57.7% of attempts in this evaluation. The reported results do not establish that human review can be skipped, and performance may differ on a particular repository.
Which open-weight model scored highest?
MiMo-V2.6-Pro scored 45.4%, the highest result among the open-weight models listed in the source material. DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6%.
Which models were missing from the results?
As of October 13, 2026, the source material reported no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. It did not give a date for when results might be added.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
