🔍 Read the full analysis: Doctors Vs. Microsoft’s AI: Understanding The Study’s Metric on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft’s peer-reviewed Science paper reports that MAI-DxO achieved 83.2% diagnostic accuracy on 304 difficult case records, compared with 36.1% for generalist physicians using reference materials. The test used historical, text-only cases in a simulator, and the comparison was against individual generalists rather than specialists or clinical teams. A study with consenting patients alongside normal care is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians who could consult textbooks and the internet, according to a peer-reviewed Science paper published October 22. The result is a benchmark comparison, not evidence that the system diagnoses patients more accurately in clinical care: the test used historical, text-only cases and compared the AI with individual generalists, not specialists or physician teams.
The paper, titled “Sequential diagnosis with language models,” reports results on two groups of cases. On the 304 complex cases drawn from the New England Journal of Medicine, MAI-DxO reached 83.2% diagnostic accuracy. Physicians using reference materials scored 36.1%, while those without those resources scored 22.3%. On a separate set of 325 emergency department cases, Microsoft reported scores of 80.3% for the system and 46.8% for physicians. These figures are the company’s reported results; the study authors are all affiliated with Microsoft.
The benchmark tests how a system seeks information as well as whether it reaches a diagnosis. A case begins with a short summary, and the AI or physician requests additional details, questions or tests from a model holding the fuller case record. The researchers score the final diagnosis and estimate the cost of tests. MAI-DxO coordinates a language model acting as a panel of virtual physicians, which suggests diagnoses, requests information and challenges proposals over several rounds.
The study also reports that GPT-5 alone scored 78.9% on the difficult cases, before the MAI-DxO orchestration system was added. The source account says that the system added about four percentage points and reduced estimated testing costs, while GPT-5’s estimated test cost was $8,409 per case. These are benchmark estimates, not observed costs from treating patients. The published account notes that some methodological details are drawn from an earlier preprint and may differ from the final Science paper.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Say About Diagnosis
The results matter because diagnostic AI is often judged on fixed-question tests where all the relevant information is already supplied. This study instead examines which information a system asks for and how it balances further testing against estimated cost. That makes it a useful test of a different capability: managing a diagnostic sequence rather than simply selecting an answer from a complete case summary.
But the accuracy gap should not be read as a head-to-head measure of AI against the doctors who typically manage complex cases. The comparison group consisted of primary-care physicians and hospital generalists working alone. According to the source account, the case records were selected for difficulty and would commonly involve specialists or clinical teams in practice. Those groups were not tested. The results therefore indicate performance on this benchmark, not a demonstrated replacement for doctors or proof of improved patient outcomes.
The cost component could also be relevant to hospitals and insurers, but it remains an estimate based on test prices mapped to billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit cost. Real spending depends on local prices, workflows and patient circumstances. For patients, the key distinction is between performance on simulated records and safety or effectiveness in care, which this study did not establish.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
The paper appeared in Science, volume 394, pages 423–428. It was received in April and accepted in August 2026, according to the supplied source. An earlier version was posted as a non-peer-reviewed preprint in June 2025 and reported a more dramatic difference, with an earlier AI model scoring 80% against 20% for doctors. The later paper’s reported figures and comparison arms should be distinguished from those earlier results.
The preprint describes 21 generalist physicians from the United States and United Kingdom: 17 in primary care and four hospital generalists, with a median of 12 years’ experience. They could not use search engines or language models. The physicians assessed a held-out subset of 56 cases, while the AI was assessed on all 304. The Science results, as summarized in the source, include an additional open-book physician group, with access to textbooks and the internet.
Several features shape how to interpret the benchmark. A model judged diagnoses against the published final diagnosis on a five-point scale; physician re-grading of a sample showed good but imperfect agreement. When a requested test result was absent from a case file, a gatekeeper model generated a synthetic result. The preprint also says test costs are first-order estimates. The NEJM set contained no healthy patients or benign explanations, limiting what the test could show about false alarms.
““more than twice as accurate as physicians using textbooks and the internet on complex cases””
— Microsoft, in its blog post, “Building Medical Superintelligence”
As an affiliate, we earn on qualifying purchases.
Limits Beyond the Case Records
The study does not establish how MAI-DxO performs with real patients, physical examinations or imaging. It also does not show whether using the system changes treatment decisions, reduces missed diagnoses, prevents unnecessary tests or improves health outcomes. Those questions require evaluation in clinical settings.
There are further limits to the reported comparison. The physician group was small and made up of generalists; specialists and multidisciplinary teams were not included. Doctors and AI were assessed on different numbers of cases, according to the earlier preprint details in the source. The case collection also lacked healthy patients and harmless explanations, so the benchmark could not measure false-positive diagnoses in a representative mix of patients. It is not yet clear from the supplied account whether every detail of the preprint’s methods carries over unchanged to the final paper.
Peer review evaluates a paper’s methods and reporting, but it does not mean an independent research group has reproduced the results. The supplied source identifies the findings as Microsoft’s own and does not report an external replication. The accuracy figures and cost estimates should be understood within those limits.
As an affiliate, we earn on qualifying purchases.
Testing MAI-DxO Alongside Care
The next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the supplied source, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence on how the tool behaves with live clinical information, but no results from that study are available here.
Readers will need details about the study design, including how many patients will take part, what conditions will be assessed, how clinicians will use the system, and how safety and diagnostic errors will be measured. Any findings will also need to be compared with usual care and assessed for effects on patient outcomes, not just benchmark accuracy. Until such evidence is reported, the Science paper establishes a result on selected simulated cases—not the system’s performance in routine medical practice.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s study find?
Microsoft reported that MAI-DxO achieved 83.2% accuracy on 304 difficult case records. Generalist physicians scored 36.1% with reference materials and 22.3% without them, according to the paper’s results as summarized in the source.
Did the study test the AI on real patients?
No. The benchmark used historical, text-only case records in a simulator. It did not include patient examinations or establish how the system performs in clinical care.
Who were the doctors in the comparison?
The comparison involved generalists, including primary-care physicians and hospital generalists. The source says they worked alone; specialists and physician teams were not included.
Does peer review mean the results have been independently confirmed?
No. The paper was published in Science, but the supplied account says the results are Microsoft’s and does not report an independent replication. Peer review is not the same as reproduction by an outside research group.
What is the next planned test?
The source says a prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with MAI-DxO alongside their usual care. The source provides no results from that study.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
