Add preferred source
Ramakrishnan Lokanathan, an AI safety researcher at Lawrence Berkeley Lab who red-teams NVIDIA’s Nemotron 3 models, says the reason most companies get nothing from generative AI is not weak models but weak evaluation. His contrarian rule: if your test scores 100 out of 100, the test is wrong — because current models top out at 85 or 86 on anything hard. He lays out a five-part framework: define the most ambitious version of success, grade every step of a workflow rather than the final answer, combine automated checks with human review and adversarial stress tests, keep hardening tests until they fail again, and feed real customer failures back in. He cites Google Assistant losing the ability to set reminders after its Gemini migration as proof that a smarter model can regress. He also introduces eSlams, an open-source package that makes AI agents play roughly 50 games across 90 providers to expose decision quality static benchmarks miss, with a “Grand Slams of AI” tournament planned for February. The stakes are concrete: MIT counts 95% of organizations as seeing no returns on generative AI, and Lokanathan argues evals are how that gap closes.
Key Elements
If your AI system aces every test you throw at it, the problem is your test — not a triumph of your model. That is the counterintuitive claim at the center of a framework laid out by Ramakrishnan Lokanathan, an AI safety researcher at Lawrence Berkeley Lab who red-teams NVIDIA’s Nemotron 3 models, speaking on the AI Engineer podcast.
His reasoning is simple arithmetic. On any model eval today, people top out around 85 or 86 out of 100; nobody reaches 100. So a test that someone finishes perfectly has stopped being a stress test and become a participation trophy. He applies the logic to the SAT: if everyone scores in the 99th percentile, the exam itself has to change.
“If you have an eval that is scoring 100 of 100, your eval is wrong, right? And you are not ambitious enough, and you’re not testing enough.”
This matters now because the money is already spent. Lokanathan cites the MIT-GenAI divide finding that 95% of organizations see no returns from generative AI, and his diagnosis is that they skipped the eval discipline that would have turned a demo into a product. “More often than not, I think evals are broken because we overcomplicate it and we don’t really understand,” he said. He speaks from operating experience rather than vendor interest: he started building on the GPT-3 API in September 2022, two months before OpenAI released the ChatGPT research preview, pivoted repeatedly, built a pet-care AI system monetized through e-commerce and recommendations, and was acquired in India. He also works with the London Business School data science and AI initiative and says he sponsors no vendor.
Start by imagining the best possible outcome
Lokanathan’s first rule is to define the most ambitious version of success before writing a single test: picture a world of AGI and ask what the agent would do at its very best. That sets the scope for everything downstream. He frames the design questions as “what must happen? And what must never happen? And when should it stop?” and extends them into the other WH-questions.
He is equally firm that no single benchmark measures a system’s ability. Being strong on Humanity’s Last Exam or SWE-bench is one signal, not a verdict — the same way the SAT is one indicator of college readiness in the US, while college admissions in his native India still hinge on a single examination, exactly the over-indexing he warns against.
“No eval or no test is a measure of someone’s complete abilities, and that should always be clear going into that.”
The final answer can look perfect while the pipeline quietly fails
His central technical argument is that grading only the start and end of a task hides failures in the middle. He offers an image-editing example: the user asks for only the background to change, and the output does deliver a cream background — but the image now has two handles. Judged start-to-end, the job looks done. Judged step by step, it failed.
The rule scales to long agentic workflows, where each step must be checked for its input, its output, and its expected outcome — and each test must match what that step actually does. If a step’s job is making tool calls, the eval for that step should measure tool-call accuracy, not some unrelated capability.
One score is not enough: build a system
The era of measuring a single capability — coding, or writing — is over, Lokanathan argues, because today’s systems are agentic, persistent, and must do many things at once. His holistic system has three moving parts.
| Component | What it requires | Why it matters |
|---|---|---|
| Automated checks in CI | Eval questions run against the agent on every production deployment | Stops evals from being a one-time score taken when the model was chosen |
| Continuous human review | Humans confirm results make sense; agents may speed things up but never serve as the reviewer | Guards against self-referential evaluation |
| Adversarial tests | Deliberately break the workflow and put it in hostile situations | Surfaces failure modes normal tasks never touch |
He is explicit that agents can accelerate the eval process but must never be the reviewer of their own work.
Chess, poker and arcade games as an eval paradigm
The newest material is eSlams, an open- agents play games. His reasoning: getting one right answer over and over is not the same as making a good sequence of decisions, and SWE-bench-style tasks involve no continuous thinking or self-evaluation
“Getting one right answer more and more is not the same as making a good sequence of decisions.”
Board games like chess and Go have long been the best human measure of long-chain reasoning, and AlphaGo was a landmark — but it was a purpose-built Go model, and today’s general-purpose LLMs are not tested on chess or Go at all. The platform supports about 50 games across 90 providers, grouped by the capability each one probes:
- Chess and Go — planning with invisible state
- Poker — decisions under hidden information
- Cooperation games — shared goals and cooperation
- Arcade game trials — a further reasoning surface
His suggested use case is model migration: when swapping in an improved model, have the old agent play the new one at chess. He is careful not to claim the winner is the better model — rather, how each enters and plays the match reveals more, because strategy shifts, blunders must be escaped, and past moves become liabilities. A preview at eSlams.com shows a tic-tac-toe match between Fable 5.1 and GPT-6 Astra that ends in a tie; visitors can also play against the models themselves.
He frames games as one additional paradigm to stack alongside coding evals, writing evals, and Humanity’s Last Exam — not a replacement. And he is putting a public test behind the bet: eSlams will host the “Grand Slams of AI” in February, an Olympics-style tournament where models compete across games.
The Google Assistant lesson: a smarter model can be worse at its job
The most concrete failure story in the talk is Google Assistant’s migration to Gemini. Lokanathan used Google Assistant for all his reminders; after the move to Gemini, it stopped setting them, explaining that it was a language model and could not set up reminders. His reading is that Google did not eval what already worked before shipping the new product.
“A model that is extremely better than the old one could very well be incapable of something the old one did very well, for example, setting a reminder.”
The lesson: keep regression cases, known tasks, and past failures in the eval flow permanently, paired with fresh challenge cases so the model is always pushed. It is an old software tradition — turn real-world failures into future tests.
Until scores hit 85 or 90, then make the test harder
Lokanathan’s headline rule of thumb gives teams a trigger for when their eval suite has gone soft, and it connects directly back to his opening claim about perfect scores and the 85-to-86 ceiling on current models.
“If you feel that your model is getting to an 85 or a 90 on your eval test, it’s time that you better your eval and push the frontier.”
Below that line, a rising score is progress. Above it, a rising score is a signal that the test is no longer measuring anything that matters — and the job becomes designing harder problems, not celebrating the number.
The customer is the last line of defense
Lokanathan closes the loop with production feedback, and every failure needs a defined path into the eval system. The mechanics he describes: use observability tools to collect automatic data; give users a simple thumbs-up or thumbs-down to report problems; then apply classic QA — reproduce the issue, find where the model failed (at the tool call, say), identify which part of the eval suite missed it, and add the case to the pipeline.
Customer feedback and human review are, in his framing, the two ways to reach a genuinely holistic eval, because the last 5% of edge cases are the ones neither the model nor an existing eval framework will catch on its own. He also flags a specific frontier he encountered: the Parallel search model index, and how well models fetch results from sources like the AI search index — relevant for anyone evaluating a model that needs real-time data. His broader advice is to keep hunting for new arenas and paradigms of evidence.
Five rules — and an uncomfortable forecast
The talk consolidates into five rules, all aimed at one idea: test the job, not just the answer.
- Define the most ambitious success — aim at the job, not the score.
- Test the whole workflow at every step, with tests designed for what each step does.
- Gather evidence holistically across as many paradigms as possible.
- Evolve the tests over time, covering past regressions and future challenges; when scores hit 85–90, make the test harder.
- Learn from reality — feed real user feedback and human review into the workflow.
“Test the job and not just the answer.”
Then comes the twist: Lokanathan predicts evals will not be relevant in a decade, because as model capability keeps improving, teams will not need as much eval as they do now. Today they are the mechanism for closing the gap that leaves 95% of organizations with no returns; in ten years they will be mere validation. The unresolved tension is that this forecast sits awkwardly against his own fourth rule: if improving models require harder tests, more capability should demand different evals, not fewer. The eSlams bet — that games reveal decision quality that static benchmarks cannot — is the part of the argument the market can watch play out, with February’s Grand Slams event serving as a public test of whether that paradigm produces signal anyone trusts.
For investors, the takeaway is that the eval layer is becoming a discipline of its own, and companies that treat it as an afterthought are exactly the ones likely to show up in that 95%. The tools are increasingly openbut the framework is what matters: measure the whole job, harden the test until it breaks, and never trust a perfect score
Full content available at:If Your AI Scores 100%, Your Eval Is Wrong – Ramakrishnan Lokanathan, eslams.com
- AI Spending Concerns Intensify as Nasdaq Plunges More Than 2% at Open; Intel Craters 25%
- Valve Says It Won’t Ban AI Games on Steam, Will Sharpen Discovery Instead
- Anthropic Engineering Lead Drops Coding Interviews: Claude Can Solve the Problems, So Hiring Now Looks at Portfolios
- Amazon Tells Engineers to Quit ‘Tokenmaxxing’ as AI Usage Metrics Backfire
- OpenAI Says Its AI Toppled a 27-Year-Old Computing Assumption; the Authors of the Theory Are Checking the Work
