Mercor's CEO: Why agents that pass tests fail in production

Zero-Shot Learning is a podcast about how AI gets built, secured, and deployed. Hosted by Nancy Wang, 1Password CTO, and Dev Tagare, Senior Director of Engineering at Google, it’s a builder’s view of the architecture and the complex decisions it takes to ship with AI. Adarsh Hiremath, Co-founder and CEO of Mercor, joined Zero-Shot Learning to talk about why we need to rethink how we measure agentic performance and the infrastructure we can build to get there. Mercor is an AI-powered hiring platform that organizes human expertise to train AI. Their talent assessment engine connects many of the leading AI labs and frontier models with specialized experts who evaluate and train the next generation of LLMs and autonomous agents. Out of that work came the APEX benchmarks, an evaluation suite that measures whether frontier AI models and agents can perform economically valuable work. In this conversation, Adarsh shares how the framework Mercor has built can help CTOs answer whether their agents actually do what they're supposed to. What do people get wrong about testing agents? "I think there are a lot of enterprise teams that move to production without fully thinking through whether the agent is calibrated to the specific use case," Adarsh said. He pointed out that many teams measure an agent's success by whether it delivers the right result, not by how it gets there. "You could have a model that just answers correctly the first time, but it's making all the wrong decisions along the way," Adarsh said. "Then when you adapt it to a slightly different context, all of a sudden you've got an agent that's totally broken in production." Testing is also complicated by AI’s tendency to “cheat” on tests rather than arriving at the correct answer on its own. In February 2026,OpenAI stopped reporting SWE-bench Verified scores after finding that all frontier models could reproduce their benchmarking test answers verbatim. The test was sourced from open-source training repositories, and because the models had seen the answers before the test ran, they recalled them rather than solving for them. What does a production-grade eval measure? As OpenAI’s Agent Security Lead, Fotis Chantzis, discussed when he appeared on the show, agents are not stable, predictable actors. They change with model updates, backend prompt revisions, and even through the inputs users provide at runtime. To account for the non-deterministic nature of agents, production-grade eval suites need to be representative of the work the agent will do and evaluated continuously to maintain effectiveness. Without a comprehensive eval suite monitoring the agent's task, trajectory, and output, there is no quantitative way to know if your agent is working or if you are helping it. If the task tells you whether the agent is working on the right problem, trajectory measures how it got there, and the output evaluates whether it succeeded. People often view investments in evals as a one-time investment. I think that is extremely far from the truth. These are live investments." –Adarsh Hiremath, Co-founder and Co-CEO, Mercor Adarsh compared these ongoing investments to the tests you write for a new product feature. You would update those tests when the feature changes, and an agent’s eval suite should work the same way; it requires maintenance as the agent, the model, and the tasks evolve. LangChain's State of Agent Engineering survey found that only 37% of respondents monitor agent performance through online evals in production, the kind of continuous measurement that catches regressions as models update and prompts evolve. And 22.8% of organizations with production agents aren't running any formal evals. Most teams have invested in watching agents and tracing their actions after the fact, but not in catching problems before behavior drifts. How do you safely deploy agents to multiple users? "It's really, really hard to roll out agents to production," Adarsh acknowledged. Gartner's 2026 Hype Cycle for Agentic AI affirms this, and notes that fully autonomous agents are not ready for most enterprise use cases, and that human oversight remains essential. In Adarsh’s experience, the secure way to deploy agents is through a staged model that limits the scope of failure at every step. Start in a personal-assistant context, where if the agent fails, only one person is affected. From there, connect it to tools in a sandbox without real data. Run your eval suite to stress-test performance before any real users get involved. When the evals clear, you can test with real users but not real data. If the agent succeeds in each stage, you can execute a phased rollout. "If the agent fails, the impact is on the order of magnitude of failing for one person, as opposed to maybe a million people," he said. "The key is not jumping the gun and following the whole process." How do you govern continuous, long-running agents? Once an agent enters production, the question changes from wheth

Excerpt shown under the publisher's RSS terms · 110 words · full text and figures are on the publisher's site