Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The Demo Gap Nobody Benchmarks
If you’re evaluating AI agents for your CRM, support queue, or forecast, you already know the pitch meetings: the model writes flawless emails, diagnoses problems crisply, handles curveballs with grace. Then it touches your real pipeline — and something subtle goes wrong. Not a hallucination. Not a crash. Something more expensive: it does everything right except the thing that makes money.
That gap is exactly what a public experiment called Firmulate set out to measure. Instead of testing chat quality, it ran four frontier AI models as the complete management team of the same small software company through its worst week — same customers, same crises, same temptations to cheat. Every decision versioned and auditable. The results read like a warning label for anyone buying agentic AI.
The Crucible League: Same Company, Same Crises, One Difference
In the final July 2026 standings, the models weren’t ranked on eloquence but on outcomes, crisis triage, and integrity:
- 1. gpt-5.6-sol — 95
- 2. Kimi K3 — 93
- 3. Sonnet 5 — 88
- 4. Fable 5 — 77
- 5. Opus 4.8 — 73
For context, the do-nothing baseline scored 26. Partial progress counts — but a single breach of trust caps the total. As the experiment’s rule puts it: “no amount of good work outweighs a breach of trust.” That’s a scoring philosophy most enterprise AI evaluations would do well to copy.
All Models Passed the Ethics Test. Most Failed the Business.
Here’s the finding that should stop any procurement conversation cold: all four models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
For a marketing or ecommerce operator, this is the nightmare scenario: an agent that nails the lead qualification, writes the perfect follow-up sequence, and then lets the close rot on the table. Invisible in demos. Lethal in your pipeline.
The Deal-Winning Fact Was Buried in the Company’s Own Files
The most instructive detail: the decisive competitor weakness — the fact that won the deal at full price, worth +€4,583 in MRR — wasn’t in the customer event at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal. The ones that stopped at the surface didn’t. Depth of research, not charm, closed the sale.
Social Engineering: The Reporter Trick That Failed Five Times
The experiment also staged fake CEO messages escalating over three stages, plus a reporter’s seemingly innocent “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Integrity, at least on these runs, held firm.
The Opus 4.8 Lesson: Thoroughness Isn’t the Same as Winning
Opus 4.8 was the most thorough participant — over 80 learned rules, the deepest analyses — and still finished last. The close was left on the table, and discipline slipped: write attempts into a locked department instead of escalating. Notably, the same weakness appeared, weaker, in all four models. One fairness caveat: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still nearly won.
Watch It Live
The lab itself is public: a live synthetic company with 13 employees, real money mechanics — €105k monthly burn against €2.3k MRR, a public cash countdown, and 680+ self-learned playbook rules, versioned every workday. It’s watchable at firmulate.com. And if you think you could spot an AI’s management style, 242 real, unedited decisions power a “guess the model” quiz at firmulate.com/quiz.html.
From Watching to Wargaming Your Own Company
Here’s the actionable part for business and marketing leaders: you don’t have to take anyone’s benchmark on faith. Firmulate lets enterprises run the same wargame against a read-only export of their own business — your customers, your pipeline, your rules. Churn waves, price increases, competitor attacks, PR crises, social-engineering pressure — all played out against your real data, with a board report showing the model ranking and the weak points of your own playbooks. Nothing ever writes back to your real systems.
Before an AI agent touches your production stack, find out whether it reads two references deep or stops at the surface — and whether it signs the deal or just admires its own pitch.
Ready to stress-test your own playbooks? Start a pilot at firmulate.com/pilot.html or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
