Braintrust's Ankur Goyal: Code review doesn't cover prompts

Zero-Shot Learning is a podcast about how AI gets built, secured, and deployed. Hosted by Nancy Wang, 1Password CTO, and Dev Tagare, Senior Director of Engineering at Google, it’s a builder’s view of the architecture and the decisions it takes to ship with AI. Ankur Goyal, Founder and CEO of Braintrust, which bills itself as “the AI observability platform,” joined Zero-Shot Learning to talk about the problem every team shipping AI eventually faces, you can build something that works and then watch it quietly become something that doesn't. Braintrust sits in the iteration loop for AI products, helping teams trace production events, turn behavior into eval datasets, compare prompt or model changes, and catch regressions before they reach users. For teams building agents, quality depends on whether the system’s behavior remains useful and safe as prompts, models, tools, and user inputs change. In this episode, what begins as a conversation about evaluation frameworks and production feedback loops reveals a security gap many teams may leave open: prompt changes are behavior-shaping production artifacts. They can change what agents do, what data they surface, which tools they call, and how they use access and credentials. Why don't prompt changes go through the same security review as code? "I think enterprise developers are more comfortable iterating quickly on prompts than they are changing the underlying code," Nancy said. Before code is sent to production, a developer commits to version control, opens a pull request, waits for peer review, passes automated security scans, and gets sign-off before anything merges. The process documents who changed what, when, and why. By contrast, prompt changes often live outside the codebase, stored in a database row, prompt-management tool, or platform dashboard that a product manager or operations team can update directly. There are often no pull requests for review, no security scans, and sometimes even no recorded change a security reviewer would see. Many behavior-shaping changes don't register as prompt changes at all. A developer adjusts how a variable is injected, adds a retrieved context source, or modifies a template's structure, and nothing in the codebase signals that the agent will now behave differently. I am, almost to a scary amount, frequently surprised by what the ramifications of random prompt changes are.” –Ankur Goyal, Founder and CEO, Braintrust When a product manager updates a customer support agent's system prompt to be more helpful, the underlying code hasn't changed, nor have the credentials in use, or the access policy that applies to the agent. Without a dedicated review process, that change slips past security entirely. The agent may now answer differently, surface different data, call tools it didn't before, or treat a workflow as in scope that wasn't before. The credentials and access policies haven't changed, but the system that appeared compliant on paper is now behaving differently in production. How do prompt changes affect agent access? Agent behavior and access are intertwined. Agents depend on prompts that shape how they interpret tasks, tools that determine available actions, credentials that grant access, and models that affect how consistently they follow instructions. Any change to that execution context, the prompt text, the model version, parameter settings, or retrieval configuration can shift how the agent behaves without changing a line of underlying code. When a prompt changes, there are usually no alerts to fire and no tests to fail against. When LLM-integrated systems are haphazardly designed, a slight change in a natural language prompt can cause an agent to overreach without producing an error. When this happens, there is no way to detect if it exposed data or used a credential unintentionally. Unlike traditional code reviews, the risk lies in whether the system’s behavior stays within intended boundaries. Because agents work behind the scenes across SaaS apps, APIs, internal systems, databases, and dev environments, they rely on non-human identities, service accounts, and shared secrets. If the credentials they use are long-lived or over-permissioned, a prompt change can quickly become more than a quality issue. Entro Labs’ 2025 NHI and Secrets Risk Report found that 1 in 20 AWS non-human identities held full admin privileges, while only 38% of NHIs had been active in the previous nine months. That's the access exposure within which a prompt change can create risk. What does the minimum release gate for a prompt change look like? When Nancy asked Ankur what a minimum release gate for a prompt release should look like, his answer hinged on a maturity model. The minimum release gate should require humans to review representative examples before a prompt change ships. Then, a team translates that feedback into scoring functions, scales testing from a few examples to hundreds, and audits the highest- and lowest-scoring o

Extras afișat conform termenilor RSS ai publicației · 113 cuvinte · textul integral și figurile sunt pe site-ul publicației