SOP-Bench: A new benchmark for evaluating AI agents on real business procedures

A standard operating procedure, or SOP, is the written set of steps an organization follows to correctly complete an important piece of routine work the same way every time. Almost every industry runs on SOPs. A hospital uses one to register a new patient, a logistics team uses one to decide whether a shipment qualifies as hazardous, a bank uses one to verify a new business customer, and a trust and safety team uses one to decide whether to remove a piece of content. SOPs carry an organization's hard-won knowledge, its compliance rules, and its decision logic in a form that any trained person can adopt and follow. As a result, they keep operations consistent and safe across different employees, shifts, and sites. SOPs are hard for AI agents to execute because they look cleaner than they actually are. A real procedure asks the reader to interpret instructions that were never fully spelled out, to draw upon knowledge that everyone in the field already shares, and to make judgment calls as conditions change. An excerpt from a patient intake SOP. Steps four and six tell the operator to verify…