🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Standing: What Its Pass Rate Leaves Out on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai says GLM-5.3-FlashX-Legal scored a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark; Vals AI’s archived leaderboard records 19.17%, seventh among 81 models. The score means the model met every criterion on about one in five tasks, while its individual-criterion pass rate was 92.5%. Those figures do not establish that its legal work is reliable without human review.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh of 81 models in the archived results. The score means it met every required rubric criterion on roughly one in five tasks—not that it got only one in five individual legal questions right—and the distinction matters to firms weighing whether to test the system for professional work.
Z.ai describes the model as optimized for legal research, document understanding, reasoning and task execution, including contract review and drafting. Its developer documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning in “tens of thousands” of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services for laws, cases and company information.
The company’s benchmark statement gives a 19.2% all-criteria pass rate and says that is 12.5 percentage points above the base model. Vals AI’s archived leaderboard lists GLM Legal at 19.17%, versus 6.67% for GLM 5.3 Flash, the base model. That difference is 12.50 percentage points in the same table. Z.ai also calls the result “state-of-the-art performance among open-source models in this evaluation”; the supplied materials do not define the comparison set, and they report no published weights for the legal model.
The leaderboard reports a separate 92.5% criterion pass rate for GLM Legal. In the archived version, Vals says a criterion counted as passed only when both of its AI judges passed it. The benchmark’s task score is more demanding: a task passes only if every required criterion passes. A missed deadline, citation or other rubric item can therefore make the whole task fail, even if many other parts are correct.
Why One Miss Can Fail a Task
The gap between the 92.5% criterion rate and the 19.17% task rate shows why a single headline score cannot describe all aspects of performance. The first records performance on individual rubric items; the second asks whether a complete work product satisfied every requirement. For legal work, that distinction is consequential: an answer may be broadly useful yet still omit a deadline, misstate a fact or mishandle a citation that matters to a client or matter.
The result may help legal teams compare systems and decide what to test, but it is not evidence that the model can safely handle matters without supervision. The benchmark measures a defined set of tasks under a particular grading process. It does not, by itself, establish accuracy across every jurisdiction, practice area, client record or live workflow. Firms considering procurement still need to assess output verification, data handling, access controls and the responsibilities of lawyers reviewing the work.
Cost is another part of the comparison, not a measure of quality. Vals lists GLM Legal at $0.15 per test in the archived leaderboard, while Z.ai’s pricing page lists $0.37 per million input tokens and $1.25 per million output tokens. Those are different pricing measures, and neither establishes the total cost of a deployed workflow, including human review and connected services.
Popular right now
Samsung 75-Inch M80H Series Mini LED 4K Smart TV (2026)
Hisense 85 U7 Mini-LED ULED 4K UHD Smart Google TV (85U7SG, 2026)
Shark HydroVac MessMaster Cordless Wet Dry Vacuum and Mop Combo, AW261
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas, each built around an instruction, a client matter and a requested work product. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria covering items such as facts, conclusions, citations, deadlines, dollar amounts and formatting.
Harvey’s all-pass rule is central to interpreting the result: a task counts as complete only when every criterion passes. The leaderboard uses a private holdout set. Vals says its holdout contains 120 tasks and disables internet access during testing. Two language-model judges—GPT 5.5 and Claude Sonnet 4.6—grade each submission, and the reported task pass rate is the average of their results.
The archived Vals Version 1 leaderboard, last updated October 18, placed GLM Legal seventh of 81 models. Its reported task rate was close to Gemini 4 Argon’s 19.58%, below several entries above it, and higher than the listed 6.67% for the GLM base model. Harvey’s May 26 baseline had a 7.1% best all-pass result among the frontier models it tested. That earlier run used a different evaluation and is not directly comparable, but it illustrates how difficult the benchmark’s end-to-end threshold can be.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.””
— Z.ai
What the Benchmark Cannot Establish
The leaderboard does not establish how often the model’s mistakes are minor or serious in actual legal matters, nor whether the same performance holds across different types of client work. The private holdout contains 120 tasks, and the results depend on its task mix, scoring rubric and two AI judges. The supplied information does not provide task-level results or a breakdown of errors by practice area.
The source article describes a four-task spot check in which the legal model cited rulings the author could not find. The author says OpenAI and Claude models in that small check invented no rulings, but two Claude models cited a real ruling for the opposite of what it held. This is an account of a limited spot check, not a benchmark finding or a basis for general comparisons among the systems.
It is also unclear from the supplied materials what Z.ai’s “open-source” comparison set includes. The documentation does not name the benchmark operator, while Z.ai’s October 13 blog post names Vals; the supplied source says no published weights for GLM Legal were found. The materials also do not specify how the model or its connected services handle client data, or what contractual and security controls apply.
Questions Before Legal Deployment
For legal teams, the immediate next step is to test the system against their own representative work, with lawyers checking every output and citation before use. Any evaluation should record the kinds of errors that matter in practice, not only aggregate pass rates, and should assess whether connected case-law, legislation and company-data services return accurate, current material.
Before a procurement decision, firms will also need answers from Z.ai about client-data processing, retention, access and security, as well as the review controls expected for generated drafts. The supplied announcement and documentation describe intended workflows and benchmark performance, but do not settle those operational questions. Further leaderboard updates or fuller disclosures from Z.ai could clarify how the model compares within a defined model set and how its evaluation was conducted.
Key Questions
What does GLM Legal’s 19.17% score mean?
On Vals AI’s archived leaderboard, the model met every required criterion on 19.17% of benchmark tasks. A task with even one missed criterion counts as a failure, so this is not the share of individual questions or rubric items it answered correctly.
How can its criterion pass rate be 92.5%?
The 92.5% figure covers individual criteria, while 19.17% covers complete tasks. A model can pass most criteria in many tasks yet fail the all-or-nothing task measure whenever it misses at least one required item.
Did Z.ai’s model rank first?
No. The archived Vals AI Version 1 leaderboard placed GLM Legal seventh of 81 models. Z.ai described it as state of the art among open-source models in the evaluation, but the supplied materials do not define that comparison group.
Does the benchmark show the model is safe for legal practice?
No. It reports performance on a defined benchmark, not a guarantee of reliability in live legal work. The source materials do not establish error severity across real matters, data-handling safeguards or whether outputs can be used without lawyer review.
Source: ThorstenMeyerAI.com
