AI Agent Regression Testing After a Prompt or Model Change

A ~author/family-latest alias always resolves to the newest concrete model in a family. That’s convenient in production and a problem in a regression test, because the model can change between runs without any change in your repository. Our latest model resolution docs describe the mechanism and recommend a concrete model slug when you need a fixed version for reproducibility. This guide covers the locked case set and the behavioral contract, then the model-swap case in detail. Regression testing an agent means re-running a locked set of cases every time a prompt, model, tool definition, or retrieval setting changes, then checking the result against a written behavioral contract. Each case carries a structural assertion about which tools the agent called with which arguments, and, where a policy exists, a hard invariant the agent must never break. Lock the case set. Every time you reword a case, you break comparability with every run that came before it. For a model swap, hold the prompt, tools, cases, judge, and inference parameters still and vary only the model. Use a concrete slug such as anthropic/claude-fable-5.1 rather than an alias that resolves to whichever version shipped most recently. Read the baseline column before the candidate…