You Can't Iterate to Trustworthy AI Code Without Understanding

As the AI industry races towards recursive self-improvement (RSI), an important question arises: to what extent is human understanding of the code AI generates required for reliable and safe automation? The question is sharpest for safety research, where undetected failures can produce flawed safety cases that allow deployment of unsafe AI. This is a crux for how quickly RSI can proceed. In this article I argue that iterating on the outcomes from automatically generated code without human understanding is inadequate. This has implications for how much automation can be done safely and for what needs to be measured to mitigate risks. In "Crisp and fuzzy tasks" Leike laid out the case that automating fuzzy tasks is harder and that it is the fuzzy tasks that make automating alignment research difficult. In introducing the concept he states that tasks like solving coding competitions are crisp because it is easy to verify the solutions. More recently, discussions have subtly shifted the frame to suggest that automating coding is a crisp task, e.g., in "Automated Alignment is Harder than You Think" Bowkis et al. give as an example of crisp tasks "writing code for an experiment." Note the shift: competition code is crisp because there's an automated checker but experiment code got classified as crisp because it resembles this. But we already have evidence that implementing experiment code from a specification (spec) is hard to verify. For example, "Measuring coding agent misalignment in the wild" showed a meaningful rate of two kinds of failures from automated coding: overselling and monitor evasion. E.g., they found examples of overselling in 34.7% of SWE-chat transcripts and 5.7% of Transluce internal transcripts. Bowkis et al. acknowledge that "While agents perform tasks that appear crisp, those tasks contain many judgement calls about details." The fuzziness and judgment calls are not limited to research questions; they are core to verifying whether the code accurately implements the intended semantics. Boris Cherny, the creator of Claude Code, published steps of automation in which later steps (3 and 4) shift away from humans reviewing code to updating specs and context for automation. Cherny notes that advancing requires "giving Claude ways to verify its own work end to end." At step 3 "'Did you read the code?' becomes 'what context was the model missing and how do we fix it next time?'" Cherny observes that "Anthropic is on step 3 and pushing toward 4. Personally, I just hit level 4." Cherny leans on automated review and sandboxing at every step and warns adopters to establish trust in iterative loops before advancing to more automation. The approach rests on the assumption that trustworthy iteration loops can be used effectively to discover and correct coding mistakes. In this piece I argue that verifying code implements a spec for real world software is a fuzzy task, and in high stakes domains like safety research and indeed model development in general it is important to treat it that way. As a consequence of this, safety researchers and frontier AI labs should verify understanding of the code that implements specifications and not succumb to the siren call of automating code with only reference to a spec. Understanding code is currently necessary, but not sufficient for safety. Verifying Real-World Coding is Fuzzy A simplified model of coding is that there is a well-defined spec that gets faithfully implemented by a developer. However, real-world coding is an iterative process of discovering complications, testing results, responding to tests, and clarifying requirements. No spec fully captures all implicit requirements. Often non-functional requirements (e.g., what level of security, performance, and reliability are required) are specified at a high-level and the inevitable tradeoffs among them are hard to specify without reference to detailed design and implementation considerations. In principle, a research spec that is written in a formal language can be verified automatically and would be a crisp task. For theoretical research, use of Lean has made this a practical reality. However, most AI research, including safety research, is empirical where specs are not amenable to formalization. In a research domain, requirements for correctness can include numerous properties that are hard to predict in advance: numerical stability, class imbalance and other unexpected data properties, distributional assumptions, unexpected results. Understanding the details of what is being implemented and what is being encountered is a core part of correct coding and inevitably requires refinement of specs based on research taste. An important output of coding is to surface the correct questions and issues for feedback (indeed this is an important part of mentoring new researchers). In practice, verification of specs is far from being automated. For reference, note that even the leading spec-driven development tools (inc