Scaling security reviews at 1Password: Building an AI-powered pipeline

The engineers here at 1Password are always working to improve our products. With all the active development to introduce features, fix bugs, and enhance the overall user experience, numerous code changes go into every release. We strive to ensure each iteration is better than the last and that new code doesn’t introduce vulnerabilities. A key part of this process is our Product Security (ProdSec) team’s review of all code changes that may have security implications. In the past, security engineers gathered on calls several times per week to go through all the PRs in the queue that required ProdSec eyes. While incredibly important, this review process was arduous and consumed countless people hours every month, especially when the engineers were flagging the same patterns over and over. And our team did this for years. These manual security reviews worked when 1Password was a smaller company with one product. But as 1Password grew in size and expanded its product line, the number of PRs increased—and that began to grow by orders of magnitude as engineers adopted AI-coding assistants. One thing remained constant, though: There are still only 24 hours in a day. It was a process that just couldn’t scale. Over the past year, it became clear we needed a solution. We tested a few popular third-party tools that use artificial intelligence (AI) to enhance the traditional static analysis process (SAST) used throughout the industry. They functioned okay from a general security perspective but we knew we could do better. We believed we could use AI models, along with a 1Password-specific knowledge base, to significantly reduce the time and effort our team dedicated to security reviews. A few days later, we had an idea and a great moniker: SAGE (Security Analysis Guidance Engine). 🌿 Analyzing our history to guide our engine SAGE was an ambitious hypothesis but we had all the information we needed, we just had to compile it. It started with a script. We gathered nearly 9,000 pull requests that spanned over five years of ProdSec code reviews. The reviews covered our entire codebase: Rust, Go, Kotlin, TypeScript, and Swift. We fed those reviews into an LLM to deduplicate comments and cull any superfluous notes (like thumbs-up emoji and “Looks good to me!”) and ended up with 8,343 reviews that consisted only of the ProdSec engineer’s comment and corresponding diff hunk. Finally, we gave those reviews to an LLM and instructed it to synthesize the information into rules, grouped by vulnerability category (and deduplicate accordingly). We (real live humans) reviewed and edited the output, and went into v0 with 171 rules across 16 categories like authentication, cryptography, and logging. SAGE v0 was deliberately modest. First, we ran it locally against PR diffs downloaded directly from GitHub. SAGE made a single call to the LLM with a relatively minimal prompt, instructions for structured output, the PR diff, and a copy of the ruleset. Our local tests were pretty successful and, after posting SAGE’s comments on the live PRs, we received positive feedback from reviewers on the team. At that point, it was time to try SAGE in the production pipeline, so we deployed it as a GitHub Action using an existing internal GitHub Actions framework as a starting point. As a GitHub Action, SAGE v0 posted its findings on the scanned PRs with comments keyed to the specific lines of code. It also maintained an activity log: A persistent PR comment updated on each push that tracked scan history (new findings, resolved findings, errors). Reviewing engineers could react to the comments with a thumbs up if they agreed with the finding and thumbs down if they deemed it a false positive. SAGE really began to show its worth when it scanned a PR that touched a lot of cryptographic code. It identified 6/6 true positives, including findings initially missed by our human reviewers. And it did all this for an average token cost of $0.47 USD per scan. But we also noticed shortcomings with the SAGE v0 implementation. Primarily, we were asking one model to locate and verify findings in the same call. SAGE also lacked any false-positive filter, had little hardening against prompt injection, and supported only a single provider. V0 proved our concept worked but it was one call/model doing everything; we needed separation and some form of objectivity. So we got back to work. A multi-model harness optimized for separate tasks We researched and brainstormed for days before we outlined a (pretty daunting) plan for SAGE v1. We started with housekeeping tasks. With access to more advanced models at this point, we ran another rule extraction and reconciled the output with the v0 ruleset. V1 now has access to 343 rules that cover 16 vulnerability categories. From that ruleset, we created a compact index that consists only of the rule ID and a one-line summary of the rule. That was the easy part. Our v1 architecture introduced two critical aspects: model/vendor agn

Excerpt shown under the publisher's RSS terms · 150 words · full text and figures are on the publisher's site