Where Did D Go? A Gap Between ARC's Motivation and Its Formalism

TL;DR: ARC's post does excellent work motivating a p(doom) estimator equal or better than random sampling; however, they evaluate p(doom) over a naive distribution of inputs, leaving them open to test-deploy asymmetry attacks. Trojan theory and cybersecurity practice suggest a lens and compare mitigation options. Context: I really admire ARC's focus here: If there will always be more deployment samples than testing samples, successful testing must compete with sampling in order to prevent deployed AI misalignment. I use this piece to focus on a risk of the metric as described, to pre-Goodhart their metric as it becomes a fruitful measure. if False: os.rmdir("/") Perfectly safe, and yet... I don't like it. If I were offered $5 to run this in my terminal right now, I probably would. But if I were similarly asked to approve it in a pull request, especially to a large many-contributor codebase, I'd say no. I know a lot about what my terminal would do right now, but the long-term risk of an unknown future refactor or confusion is higher than I'll tolerate. Even on perfectly safe code, I'd rather hedge against an unknown environment. The central mechanism for designing a trojan is an information asymmetry: the trojan attacker knows something about the deployment environment which the defender doesn't know. Perhaps it's a strict exclusion, such as the attacker knowing that deployed systems receive 4 significant figures from their sensors while the test platforms only provide 3 significant figures. Perhaps it's a smooth bias, knowing that test campaigns are generally conducted with low ping latency to US-East datacenters but deployment is done all over the world. Perhaps it's a controlled feature, where the attacker sends a command-and-control (C2) signal to invoke catastrophic behavior, a signal the defender never knew to test for. Either way, whenever the defender is deploying a tested system into a partially unknown environment, they run the risk of an attacker knowing the environment even better and exploiting that asymmetry. This is close to exactly the problem the Alignment Research Center (ARC) is trying to solve for AI; let's unpack their scheme (link). First, they describe a "catastrophe detector" C that detects if a model's output is catastrophic. Around C, they describe dueling systems: A distribution D of inputs which we optimize to push the model toward catastrophe, and the model itself M which we optimize to avoid catastrophe on the inputs D discovers. D's whole job is to play the attacker, to discover the narrow triggers and unexpected biases which would steer the model towards disaster. However, even with a very robust D, sampling rare behavior might take time proportional to the rarity - a one-in-a-quintillion failure, even with an efficient sampler, may take too long to discover. ARC's proposed fix is G_M: an estimator that uses a structural explanation of the model to decide where sampling is actually needed and where it can be skipped, such as decomposing a rare compound event into independent, far more common pieces, each cheap to estimate on its own (see the post for the details). G_M is defined as an estimator over the inputs to the model, drawn uniformly, fixed as part of the theorem statement itself; the adversarially-selected D turns into uniformly-selected inputs, and everything ARC proves about G_M is under that assumption of uniformity. ARC's post has partial responses to two of these three trojan mechanisms, and silence on the third. Against controlled features like a C2 signal, footnote 15 concedes the point directly, calling accuracy against adversarially-chosen context "too much to ask for." Against smooth bias: D's original objective already concentrates mass on catastrophe-inducing inputs, and footnote 3 additionally requires D's parameterization to be flexible enough to represent computationally intractable distributions, but D itself doesn't survive into G_M's formal definition. The closest surviving analogue, from the half-spaces work, is a believed-solved result for context vectors drawn from a restricted non-isotropic distribution, with no general solution for arbitrary covariance. Against strict exclusion, the post says nothing specific, leaving open issues like SolidGoldMagikarp to slip through underexposed gaps in the model's input space. Sitting above the trichotomy is ARC's footnote 6, identifying a representational fact about re-expressing structured input testing as uniform-noise testing with a converter baked into the model; however, the generator is itself part of the adversarially-chosen model parameters, not an opportunity to bias the distribution towards deployment; this relocates rather than closes the gap. None of which should suggest ARC hasn't made real progress. They report a working mechanistic algorithm for the intersection of random half-spaces, competitive with sampling both in theory and in practice. They believe they have a solution for random MLPs