🔍 Read the full analysis: What AI Benchmarks Reveal About Model Performance on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
AI benchmarks can show how models perform on selected tasks, but they do not establish how useful a system will be in every real-world setting. The supplied source material does not report a new benchmark result; it instead examines a broader measurement problem through GDP, unpaid work and the value users receive from AI tools.
A November 11 essay on ThorstenMeyerAI.com argues that headline measures can miss important parts of economic value, using that point to frame how readers should interpret AI performance measures. The source does not announce a new benchmark, model ranking or test result; it cites an estimate of consumer value from chatbots and a historical example of GDP’s limits.
The essay, by Thorsten Meyer, traces national income measurement to a 1934 report supervised by economist Simon Kuznets. The report found that U.S. income produced fell from $83.0 billion in 1929 to $39.4 billion in 1932, a decline of 53%. Kuznets warned that the measure covered mainly activity visible in markets and could not, by itself, establish how well a nation’s people were doing.
Meyer applies that distinction to AI, saying tools may create value for users that is not reflected in the revenue they earn. The article cites a Stanford working paper estimating that AI chatbots generated $172 billion a year in value for U.S. consumers. It does not give the study’s methodology, publication date, uncertainty range or a comparison period, so the estimate should be treated as a cited research finding rather than a confirmed measure of total AI value.
The source also points to a 2017 experiment in which the median participant would have required $17,530 to give up search engines for a year, despite paying no direct price for them. That figure illustrates a gap between market price and users’ stated value; it does not measure AI model quality or establish that chatbot users value AI by the same amount.
Infographic 01 · The sector clock, by output
China: how GDP changed shape
Share of GDP (value added at current prices) by sector, 1978 to 2025. In 2023 farming still held 22.8% of all jobs but produced 7.1% of GDP.
- Farming27.7%
- Industry47.7%
- Services24.6%
Reform starts
- Farming14.7%
- Industry45.5%
- Services39.8%
- Farming9.1%
- Industry45.4%
- Services45.5%
Services overtake industry
- Farming6.7%
- Industry35.6%
- Services57.7%
Communiqué 2025
Measured by output, not by jobs: the sector clocks of earlier episodes count workers. Jobs vs. output for farming, 2023: 22.8% of employment (year-end) vs. 7.1% of GDP.
Sources: NBS China, China Statistical Yearbook 2024, tables 3-2 and 4-2; NBS, Statistical Communiqué 2025 (28 Feb 2026).
Powered by Thorsten Meyer AI
Why Benchmark Scores Can Mislead
Benchmarks matter because they give developers, buyers and researchers a common way to compare systems on defined tasks. A score can help show whether a model handles a test set better than another model under the same conditions. But it is evidence about performance on that test, not a complete account of reliability, cost, speed, safety or usefulness in a workplace.
The source’s GDP example highlights a related distinction: a measure can be valuable while still leaving things out. GDP tracks market production, not all unpaid work or how income is distributed. Likewise, a benchmark may capture accuracy on selected prompts while missing the time people save, the cost of correcting errors, how often a system is used, or whether its output helps accomplish a real task.
That distinction affects decisions. A company choosing a model from a leaderboard alone may overlook performance on its own data, the need for human review or the costs of deployment. For readers, the cited $172 billion consumer-value estimate is not a benchmark score and does not prove that any particular model performs well. It concerns a different question: the estimated value users receive.
Popular right now
eufy E15 Gen 2 Wire-Free Robotic Lawn Mower
Shark ZU102 Rotator Pet Upright Vacuum with PowerFins HairPro & Odor Neutralizer Technology, Charcoal
Vitamix Touch & Go Advance 32-Ounce Bar Blender, White
As an affiliate, we earn on qualifying purchases.
From GDP to AI Measurement
Kuznets’ national income report was delivered to the U.S. Senate on January 4, 1934, after the Senate requested estimates during the Great Depression. The report offered a way to quantify a severe economic contraction, while its author cautioned against treating national income as a direct measure of welfare. The article says unpaid household activity was outside the market-focused estimates.
National accounting later developed into measures including gross national product and, in the United States, GDP as the headline measure from 1991. GDP counts the market value of final goods and services produced within a country. The essay’s broader point is that the usefulness of a measure does not make it exhaustive: its scope and omissions matter when people interpret it.
AI benchmarks serve a narrower purpose. They typically evaluate models on a set of specified tasks, but the source material does not identify a benchmark, model, test set or result. Its discussion is therefore best read as a measurement-focused report, not a comparison of current AI systems or evidence that one model leads another.
“Free goods are almost invisible.”
— Thorsten Meyer, ThorstenMeyerAI.com
What the AI Estimate Leaves Open
The source does not identify the Stanford working paper by title or authors, explain how it calculates $172 billion a year, or state when the estimate applies. It is not clear whether the figure covers all chatbot users, which services are included, or how the estimate accounts for differences in use and user responses.
There are also no new benchmark results in the material: no named models, evaluation dates, test conditions, scores or independent replication. Readers cannot use this article to determine which AI system performs best, whether scores have changed, or whether a reported result transfers to a particular task. The source’s argument concerns the limits of measurement, not a finding that benchmarks are useless.
The cited search-engine experiment is also not directly comparable to the chatbot estimate. It asked participants how much they would need to give up a service for a year; it did not test model performance or measure the same outcome. The two figures illustrate different forms of user value and should not be combined into a single estimate.
What Readers Should Check Next
For a concrete model comparison, readers will need benchmark reports that name the systems tested, provide the evaluation date and describe the tasks and scoring method. Results are easier to interpret when the test conditions are transparent and when scores are accompanied by information about errors, cost and performance on relevant real-world work.
For the cited consumer-value estimate, the next useful step is the full Stanford paper. Its authors, methods, sample, assumptions and limitations would clarify what the $172 billion figure measures and how broadly it can be applied. The source material does not say when further details or a new benchmark assessment will be published.
Until such evidence is available, the confirmed development is the publication of Meyer’s measurement-focused essay, not a change in AI rankings. Benchmark scores remain one input for evaluating systems; they should be read alongside the task being tested and the value or costs that the score does not capture.
Key Questions
Does this report announce a new AI benchmark result?
No. The supplied source reports no new benchmark, model score or ranking. It discusses the broader limits of measurement and cites an estimate of consumer value from AI chatbots.
What is the $172 billion figure?
ThorstenMeyerAI.com attributes an estimate of $172 billion a year in value for U.S. consumers from AI chatbots to a Stanford working paper. The supplied material does not provide the paper’s methods or time period.
Do benchmark scores show how useful a model is?
They show how a model performed on specified tests. A score alone does not establish its accuracy on every real-world task, its operating cost, reliability or value to users.
Why does the article discuss GDP?
The essay uses GDP as an example of a useful measure with limits. GDP records market production but does not capture every form of work or directly measure welfare, a distinction Meyer applies to interpreting AI measurements.
Source: ThorstenMeyerAI.com
