Temporal CTO: A 20-year shortcut to build reliable agents

Zero-Shot Learning is a podcast about how AI gets built, secured, and deployed. Hosted by Nancy Wang, 1Password CTO, and Dev Tagare, Senior Director of Engineering at Google, it’s a builder’s view of the architecture and the complex decisions it takes to ship with AI. In this episode, 1Password VP, GM of Developer and AI, Jeff Malnick sits in for Dev. When Maxim Fateev, CTO and co-founder of Temporal, joined Zero-Shot Learning, he brought a historical perspective to the challenges developers face when building agentic systems today. From vanishing state to retry storms, Fateev saw that the failures of deploying long-running agents have parallels to the problems he’s been working on for decades. Maxim joined Amazon in 2002, where he co-created Simple Workflow Service, the internal orchestration platform that became one of the most widely used services at Amazon. At Uber, he built Cadence, the open-source predecessor to Temporal, the durable execution platform, which he co-founded in 2019. Temporal now runs production workloads for OpenAI, GitLab, Lovable, Docker, and Cloudflare, and has more than 2,500 customers globally. As the industry builds agentic systems, Fateev is watching it rediscover exactly what his infrastructure was built to solve. A brief history of failures Agents become distributed systems the moment they cross a network. Every call to an LLM, every tool invocation, every write to a downstream service crosses a process boundary, and a process boundary is where distributed systems failures begin. Two common ways agents fail in production are state loss and retry storms. While working, an agent builds state, e.g. a record of which tools it called, the results it received, and how far it progressed in a task. When the process crashes, that record is gone. There is no checkpoint to resume from, no record of what was completed, no way to distinguish completed work from incomplete work. The next run starts from scratch, leaving the operator unsure which actions can be safely repeated. When an agent calls an external service and gets no response, it retries. If it does not succeed, that retry turns a short synchronous call into a long-running operation. Multiply that across thousands of agents hitting the same service simultaneously, and the load compounds, the service stalls, and the retry pressure forces an outage. It ends up becoming a self-inflicted Distributed Denial of Service (DDoS). To survive this, a system needs flow control to pace requests, queues to absorb spikes, and rate limiting to protect downstream services. An agentic system built without any of that suddenly needs all of it. And when agents crash under the strain, there is no built-in way to tell which ones were mid-task and need recovery. "All these things stack up, and I think people who started from 'synchronous Python in-memory program' are learning the hard way that these are not simple problems,” Maxim said. Engineers first encountered these challenges in the 2000s while software was becoming the primary way people do business and manage our personal lives. As everything from booking travel to routing deliveries and managing health records became digital processes, the systems supporting it had to grow from single machines into networks of interdependent services. Serving millions of users led to process crashes and lost work, retries amplified into outages, and tasks were repeated because callers retried after partial success without confirmation. Buried in software, and software problems, engineers built their way out. Over time, they developed heartbeats to detect process crashes, queues to absorb retry storms, and architected execution guarantees that preserve workflows even if the process running it fails mid-task. These solutions eventually matured into infrastructure so foundational that most developers building agents today never had to think about it. Until now, when agentic developers have to decide which parts of that history belong in their own systems. What distributed systems already solved When state loss and retry storms appear, developers reach for patterns they already know. For many, event-driven architecture is a natural first choice. Teams will wire services to communicate through shared channels, reducing direct dependencies and making it easier to add or remove components. One part of the system posts a message when something happens, and others listen and react. This approach is loosely coupled by design, so it’s flexible, and it’s a family way to coordinate work that most teams have built before. But that flexibility has a cost that becomes due when the messaging format changes. Update one, and you may not know which services depend on it or what state they have already accumulated. There is no single, visible contract between components. Instead, the contract is partly embedded in the assumptions each service makes about every other service. Maxim describes this dependency by saying, “Events ar

Excerpt shown under the publisher's RSS terms · 103 words · full text and figures are on the publisher's site