Scaling security reviews at 1Password: Solving the context and nondeterminism problems

In our last post, we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase. Our goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are huge, including our client and server monorepos. That means we have way too much information to fit within any LLM’s context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself. There was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can’t reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer. We had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn’t just know 1Password, it would finally understand it. And it would earn the name SuperSAGE. Compressing context with scaffolding As it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation; we just had to productionize it as a Go rewrite on top of SAGE v1’s model-agnostic llm.Client harness. To start, the PoC took inventory of our code structure. Any well-organized codebase is shaped like a tree: a root that branches into directories and subdirectories, all the way down to individual files. The PoC walked that tree once from the bottom up, so by the time it reached any given directory, everything beneath it had already been analyzed. This is a classic map-reduce pattern: each directory was summarized on its own (the map), and those summaries were folded upward, child into parent, all the way to the root (the reduce). This worked well for a point-in-time snapshot of our codebase, but 1Password has hundreds of hard-working engineers, so there are sections of our code that change every day. On the other hand, other sections, like our cryptography layer, are trusted, well-vetted, and rarely touched. Re-running that full bottom-up walk every night, across a codebase the size of ours, would be slow and expensive. So we chose to make it incremental: only regenerate scaffolding for directories in which the code had actually changed. But skipping unchanged directories is only half the picture. When a directory's code has changed and its scaffolding needs to be regenerated, the model hands back new prose everytime, even if nothing more than a line of whitespace was added. An LLM rarely, if ever, describes the same code the same way twice. Had we kept folding each directory's model-written summary into the one above it, those harmless rewordings would have piled up: a rephrased child summary would make its parent look changed, and that parent its own parent, all the way to the root, leaving us regenerating the whole tree every night to chase code changes that weren’t semantically different. It left another problem sitting in front of us, even for a single directory. We'd been letting the model's wording alone decide what counted as a change. We needed a way to answer one question without depending on the prose at all: Did this directory truly change? There it was: the nondeterminism problem. Working with LLM nondeterminism The easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally. At first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s text was identical to the original, we considered checking if the new summary’s meaning was close enough to the original. It sounded good on paper, but ultimately we rejected the idea because a genuinely important security change (like adding a new endpoint that’s reachable from outside the trust boundary) might only shift the similarity score a small amount, and would be indistinguishable from ordinary LLM-rephrasing noise. For a security tool, silently missing that kind of change is worse than being too cautious. Instead, we narrowed what SAGE v2 considers a change. Rather than comparing everything within a direct

Excerpt shown under the publisher's RSS terms · 124 words · full text and figures are on the publisher's site