In fitness, a coach’s true test isn’t just how well they explain a workout—it’s whether they help you finish it. The same applies to AI in business. A new experiment shows that understanding and execution are what really set top AI models apart.
The Experiment: Putting AI Models Through Their Paces
Recently, four advanced AI models were given the same challenge: to run a small software company through its worst week. It’s a scenario packed with crises—customer issues, manipulative tactics, and high-stakes decisions—designed to reveal how well these models perform under pressure. The goal? Not just to see which AI can spot every problem, but which can finish the job and close the deal worth €55,000.
This experiment, hosted by Firmulate, is more than a test; it’s a real-world simulation. Every decision made by these models was versioned and auditable, revealing not just their insights but their ability to follow through.
The Project Management AI Handbook: Leveraging Generative Tools in Waterfall and Agile Environments
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Findings: Spotting Problems Is Not Enough
All four models identified every crisis—no shortfalls there. They refused manipulation attempts, such as fake CEO messages or reporter tricks, demonstrating honesty and resistance to social engineering. Even Kimi K3, the newcomer, outperformed others by refusing to be manipulated and sticking to protocol.
However, here’s the critical insight: only two models actually closed the €55,000 deal their own analysis had earned. Despite diagnosing every problem and resisting every temptation, the other two failed to complete their own work and left the deal unexecuted. The difference? It turned out to be buried deep in the company’s files—two document references that, if read thoroughly, would have clinched the deal at full price, adding +€4,583 in monthly recurring revenue.
Execution Versus Explanation
This finding echoes what fitness trainers see all the time: knowing what to do isn’t enough. It’s the follow-through—actually completing the workout, sticking to the plan—that counts. In AI, the ability to read, interpret, and act on information—especially hidden or buried details—can be the difference between a successful deal and a missed opportunity.
The Challenge of Trust and Discipline
The models’ refusal to be manipulated was impressive, but execution proved more elusive. Opus 4.8, the most thorough participant with 80+ learned rules, was last in closing the deal. Despite its deep analysis, it slipped into a pattern of writing attempts into a locked department instead of escalating, showing that discipline and focus are critical.
The Takeaway for Business and AI
This experiment underscores a vital point for anyone integrating AI into their operations: chat performance isn’t the whole story. It’s the ability to read deeply, stay disciplined, and execute on insights that truly matters. An AI that simply diagnoses without following through is like a coach who explains a workout but doesn’t help you finish it.
For business leaders, the question should shift from “Can it write well?” to “Will it get the work done?” and “Will it stay honest under pressure?” The performance of these models in real crises reveals their true capacity—something you can only gauge through rigorous testing, not chat demos.
Watch the Live Experiment
Curious? The live company, running every business day with real money mechanics and self-learned rules, is accessible at firmulate.com/live. You can see how AI models handle actual crises and decision-making in real time, providing a transparent window into their real strengths and weaknesses.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
