Building an agentic harness that outlasts the model

Our role in Application Security at Shopify is to keep our merchant and buyer data safe. Models have been providing us with increasingly advanced ways to do that. We’ve built an agentic code review and test oracle harness that discovers vulnerabilities in our software, proves them with real tests, and provides Shopify-tuned fixes before presenting them to our developers. Over the past five months, we’ve been evaluating new security-tuned frontier models inside that harness. The best models we tested were genuinely better than our baseline frontier models at discovering findings. But they also produced more potential issues that needed to be confirmed. In one audit, a model uncovered more than 30 candidate vulnerabilities. After validation, every one was downgraded to low or medium, found to be a false positive, or reclassified as defense in depth. The models are getting better, but the most important piece of the system remains the harness. Our scanning workflows run across our most important public-facing surfaces. They produce real findings in our web applications and help us close the loop quickly by opening relevant, Shopify-specific pull requests. In this post we share our…