🔍 Read the full analysis: The Same 15 Legal Questions, 42 AI Setups, And A Score Of 8/100 on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, assessed 42 setups against the same 15 questions on German, U.S. and EU law. Z.ai’s law-focused GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the tasks; the test’s owner disclosed that Claude agents graded answers and that a Claude model ranked first.
Thorsten Meyer AI reported on November 4 that 42 AI setups answered the same 15 legal questions in its Lexassay test, with Z.ai’s law-focused GLM-5.3 FlashX Legal scoring 8 out of 100. The result raises questions about relying on legal AI for tasks such as calculating court deadlines, while the test’s owner disclosed that Claude agents graded the answers and a Claude model ranked first.
Lexassay’s questions cover German, U.S. and EU law, including deadline calculations, legal changes, court decisions and questions built around false premises. Ten questions are public and five are hidden. Models answered without internet access or tools, according to the test description, and each answer was graded against a prepared answer using weighted criteria. The published results are a spot check: the source says each model answered each task once, so small differences in scores may reflect variation rather than a stable ranking.
GLM-5.3 FlashX Legal, which Z.ai describes as optimized for professional legal workflows, scored 8/100 and zero on the hidden set. It fully solved none of the 15 tasks and gave no answer on two German-law questions, after using its stated 64,000-token thinking budget, the source reports. Its citation review recorded 21 real citations, 25 misattributed citations and one invented citation. The underlying GLM-5.3 FlashX model scored 3, according to Lexassay.
The test’s top reported scores were Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leader met every criterion on only eight of the ten public tasks, the source says. Across published runs, Lexassay says it confirmed 2,644 citations as real and found 804 that were invented or did not support the claims attached to them. These totals are the test operator’s figures, not an independently verified audit.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Deadlines Need Checking
The test highlights a practical risk: a legal answer can sound precise yet get a date or supporting authority wrong. In a German civil case, the source explains, an appeal filed late can be rejected as inadmissible. A one-day error may therefore affect a person’s ability to challenge a judgment, rather than merely produce an imperfect summary.
One test example involves two judgments served on April 27, 2027, in Munich and Berlin. Lexassay says the one-month appeal deadline falls on May 27, a public holiday in Bavaria but not Berlin. The Munich deadline moves to May 28, while Berlin’s remains May 27; the deadline for written appeal grounds is June 28 in both cases. The example shows why location and procedural details matter. It does not establish that every model or legal AI product will make the same mistake, and the results should not be treated as a general measure of performance in real legal practice.
The citation findings also matter because references are often how users check an answer. A citation may exist but fail to support the proposition, or it may be fabricated. Lexassay penalizes invented and misattributed citations, but users still need to verify cited laws and rulings in authoritative sources. The publisher recommends treating chatbot output as a starting point, not relying on it for deadlines, and consulting a lawyer when a deadline, money or an authority is involved.
Popular right now
Dyson Ball Animal 3 Upright Vacuum – Corded, 290 AW, De‑tangling Motorbar Cleaner Head, 3 Suction Modes
Ninja Foodi DZ201 8-Quart DualZone 6-in-1 Air Fryer
LG 55-Inch OLED evo G2 Series 4K Smart TV (OLED55G2PUA, 2022)
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay compares model answers with a prepared “gold answer” containing three to six weighted criteria. The test checks citations against official databases and labels them as real, misattributed, invented or not findable. Its scoring rules also deduct points for citation failures and for answers that omit a legal-advice disclaimer or recommendation to consult a lawyer. The source says the full method is published, along with public answers, gold answers and citation checks.
The test’s setup is narrower than a lawyer’s use of research software: models answered without web access or tools, measuring what they produced from the model itself. Five questions are hidden to make it harder for later models to learn the answers from public material; only scores for those tasks are published. Effort settings are tested separately, so one underlying model can appear in multiple setups.
The source also notes that a different benchmark can produce a different ordering. On Vals AI’s legal-agent benchmark, where models work with supplied documents, Mistral Large 4 ranks eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th, according to the account. The tests use different tasks and conditions, so their rankings are not directly interchangeable. Thorsten Meyer AI disclosed that it runs Lexassay and that Claude agents graded the answers; the fact that a Claude model leads the reported scores is a potential conflict of interest.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer AI, describing the Lexassay results
Limits of the Published Scores
The report does not provide repeated trials for each model and question, so it cannot show how consistently a system would answer across multiple attempts. It also tests models without internet access or tools; performance may differ in products that search legal databases, retrieve documents or use other safeguards. The source does not establish that the test represents all legal tasks or the conditions of professional legal work.
The disclosed grading relationship is another limitation to weigh: Thorsten Meyer AI runs Lexassay, Claude agents graded the responses, and a Claude model ranked first. Meyer says the method and answer materials are public so readers can inspect the scores, but the supplied material does not describe an independent replication or external audit. The figures for citation errors and model rankings should therefore be understood as results reported by the test operator.
It is also unclear from the material provided how every one of the 42 setups performed, beyond the cited examples and leading scores. The source identifies an alternative Vals AI ranking but does not give enough detail here to compare the benchmarks’ full methodology or explain each model’s differing position. The results do not show whether a particular current legal AI product, with tools enabled, would make the same errors.
What Readers Should Verify
Lexassay’s published materials give readers a way to inspect its public questions, expected answers and citation checks. Hidden-task answers are not released, according to the source, though their scores are. No independent follow-up study or response from Z.ai to the reported result is included in the supplied material, and no further test date is specified.
For users, the immediate next step is verification, not reliance: check every cited law and court decision against an official source, and do not take a legal deadline from a chatbot. Where a filing date, financial claim or communication with an authority is at stake, consult a qualified lawyer. The test measures answers under its stated conditions; it does not determine what a court will decide in an individual case.
Key Questions
What did the test measure?
Lexassay gave 42 AI setups the same 15 questions on German, U.S. and EU law. Models answered without internet access or tools, and the test graded answers against weighted criteria and checked citations.
Why did GLM-5.3 FlashX Legal receive 8/100?
According to Lexassay, it fully solved none of the 15 tasks, scored zero on the five-question hidden set and gave no answer to two German-law questions. The source also reports citation errors, including misattributed and invented citations.
Does this prove legal AI is unreliable in every setting?
No. The publisher describes the results as a spot check based on one answer per setup and task, without web access or tools. The test does not establish how every legal AI product performs in other conditions.
Can I use a chatbot to calculate a court deadline?
The source advises readers not to rely on chatbot answers for deadlines. Rules can depend on jurisdiction and specific dates; verify the applicable law and consult a lawyer if a filing deadline is involved.
Source: ThorstenMeyerAI.com
