Picking AI For News: Why My Test Winner Isn’t The Model I Run

🔍 Read the full analysis: Picking AI For News: Why My Test Winner Isn’t The Model I Run on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on news-writing tasks and selected GPT-6 Luna for production, although GPT-6 Astra scored highest. Meyer’s October 13 report says Astra earned 4.00 out of 5 in the initial English and German tests, while Luna cost about one-hundredth as much per article and performed consistently enough for his publishing workflow.

Meyer conducted the test between September 8 and October 13, 2026, using his production prompt and real news sources. The configurations included API and subscription models, free-tier options, and open-weight models running on his own hardware. Each source was turned into an English and a German article, then assessed by two AI judges: GPT-6.1 Sol and Claude Opus 5.5.

The initial test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer says short sources are typical of his workflow, making the shorter item particularly relevant. The judges assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, along with German-language quality where applicable.

In a follow-up robustness run, the stronger configurations wrote 10 more articles, mostly from short sources. Meyer reports that Astra averaged 3.93 across 12 articles, with a 95% bootstrap interval of 3.83 to 4.02. Luna averaged 3.67 in that run, with an interval of 3.59 to 3.76. Astra cost about $0.12 per article and took around 50 seconds; Luna cost about $0.0012 in the test and took about 17 seconds. Meyer says Luna’s production cost over the previous week was about $0.0013 per article at a median of 17 seconds.

Why Luna Beat the Top Score

The results illustrate a practical trade-off for publishers using AI at scale: the highest-scoring model may not be the best operational choice. In Meyer’s measurements, Astra’s stronger writing score came with a price of roughly 100 times Luna’s per article. For a service producing articles repeatedly, cost and processing time can matter alongside editorial quality, especially when the less expensive model clears the publisher’s quality threshold.

The test also points to a separate editorial risk: a model used only when the primary system fails can still publish serious errors. Meyer says two fallback models generated invented quotations or failed to produce valid output. His account suggests that publishers should test their full routing and fallback setup, not just the model normally used.

These findings are specific to Meyer’s prompt, sources, judges and workflow. They do not establish that Luna is the best choice for other newsrooms, or that Astra’s measured advantage would hold across different topics and editorial standards. The broader takeaway is narrower: model rankings from one task or benchmark may not predict performance on another.

How the News-Writing Test Worked

Meyer’s stated goal was to evaluate models on the task his sites actually perform: turning source material into a finished article. The test used the same production prompt for each configuration and included both English and German editions. An article counted only if it produced valid output after the repair step used by his publishing pipeline.

The two AI judges scored articles from one to five across several editorial dimensions. Meyer also cautioned that, with only two articles in the standard test, differences below about 0.3 points could be noise. He treated the larger robustness run as a more reliable basis for ranking. The reported scores and intervals therefore describe this test setup, rather than an independent audit of all model performance.

The comparison also found that more reasoning effort did not produce a clear improvement for GPT-6 Luna. Across six effort settings, scores varied without a consistent gain; the maximum setting cost about 12 times more and took about 12 times as long as the no-effort production setting, according to Meyer. He said the production system would remain on the no-effort setting.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Reported Results

The report describes a test run by Meyer on his own workflow; it does not provide an independent replication. The standard comparison used two source items, and although the robustness run added 10 articles for the strongest configurations, the results may not represent other publishers’ prompts, subjects, languages or review processes.

The scores were assigned by AI judges rather than reported as ratings from human editors. Meyer says both judges saw the original source, but the account does not establish how their assessments compare with newsroom decisions or reader judgments. It also remains unclear whether the relative costs and scores will persist as model pricing, versions and service conditions change.

Meyer reports that GPT-4.1 nano and gpt-oss-120b were removed from his route on October 7 after producing fabricated quotations or unusable output. The report does not give a longer-term comparison of the revised route or describe whether human review caught or corrected those failures before publication.

How the Revised Route Performs

Meyer says GPT-6 Luna now handles routine news writing after the two fallback models were removed. The next useful measure will be how the revised route performs across a larger and ongoing set of real articles, including whether its source fidelity and valid-output rate hold as topics and source lengths vary.

For readers, the report offers a snapshot rather than a universal model ranking. Meyer’s stated approach is to keep testing models against the task they are meant to perform. Any future change in his production choice would depend on whether another model improves quality enough to justify its cost and whether the full production route can maintain reliable, source-grounded output.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna produced consistently good articles at about one-hundredth of Astra’s cost. He judged that performance sufficient for his high-volume news workflow.

Did higher reasoning effort improve the articles?

Meyer reports no reliable improvement across six GPT-6 Luna reasoning settings. The maximum setting cost about 12 times more and took about 12 times as long as the no-effort setting, while score differences were within the test’s uncertainty.

What problems did the fallback models have?

Meyer says GPT-4.1 nano received low source-fidelity scores and generated invented quotations. He reports that locally run gpt-oss-120b produced no valid articles in four attempts and also invented named quotes.

Do the findings show which AI model is best for every newsroom?

No. The results come from Meyer’s prompt, sources and evaluation process. They show how the tested configurations performed on his task, not how they would rank across all newsrooms or editorial workflows.

Source: ThorstenMeyerAI.com