Automting AI Safety Research: Managing expectations

This is the first of a series of posts about automating AI Safety Research. TL;DR Automating large parts of AI safety research now seems possible. Alignment research faces three recurring challenges: measuring model capabilities and safety, developing techniques that make models safer, and establishing why those measurements and techniques should remain reliable beyond tested conditions. Automated researchers will likely accelerate the first two. For the third, arguing that a measurement or technique survives stronger optimization pressure, safe feedback loops are harder to obtain, and we should manage our expectations on the amount of progress that will be made. I present folding inference-time techniques into training as an example where generalization arguments would be needed. A priority during the transition to automated alignment should be to measure how much automation accelerates progress on actual alignment targets and furthers our understanding of the hard parts of alignment. Introduction Fable 5.1 and GPT-6-Astra have both updated me towards large fractions of AI safety research being automated in the near future. The largest reasons for my update are the improvement on ML and coding evaluations, the improvements on writing clarity compared to Opus 5 and GPT-5.6-Sol, and the internal reports from OpenAI and Anthropic. Both Anthropic and OpenAI have shared how mid-2026 models are speeding up internal research [1] [2]; increasingly, models are performing a larger fraction of the tasks that human researchers used to perform in previous years. Anthropic research staff estimated about 4x their unassisted output with Mythos Preview in March 2026, and OpenAI projects strong progress towards an automated AI researcher in about 18 months. Based on historical precedent, it's likely that the areas where automated alignment researchers will make the most progress initially will be engineering-heavy tasks, somewhat lagging on tasks that are not well represented in the RL distribution, like setting company-wide research agendas and priorities. The state of alignment in 2026 An alignment researcher faces recurring problems that mostly come down to three questions: How do we measure some property of a model? How do we change that property? Why should the measurement and the change keep working for more capable models? For many important problems we care about, our ability to evaluate and modify models is enough to create environments that let us investigate hypotheses. As models become more capable and eventually superhuman, our observability will be reduced because the experiments may become too risky to run or because the model may have direct control on what we are able to observe. In that regime where we have limited empirical evidence we will need arguments about whether a measurement or technique will keep working under stronger optimization pressure, out of distribution, or under adversarial pressure. For ergonomics I call arguments that a measurement or technique remains reliable under these changes "generalization arguments". In the literature, arguing for generalization is often part of an "alignment argument". Aggregating evidence from many different sources is also not a solved issue: Bowkis et al. point out that correlated errors could compromise alignment assessments. Arguments for folding Currently, we have a limited portfolio of techniques that allow us to align and control models. Some of these techniques are used during training where they endure high amounts of optimization pressure. Others are used during inference like monitoring, where the optimization pressure is much weaker and mostly driven by the model and environment. A concrete case that calls for a generalization argument is folding. Folding consists of adapting inference-time techniques to work during training while preserving their core properties. This is challenging because we may need to tweak a technique to resist much more optimization pressure. For capabilities, we have examples of successful folding: prompting with a few example user-assistant exchanges was folded into instruction tuning, and chain-of-thought prompting into chain-of-thought RL. In both cases the behavior the prompt used to elicit now comes from the weights, at equal or better performance. For safety, consider the case of auto-mode classifiers. These classifiers are deployed today to stop coding agents from taking dangerous or destructive actions: they read each action the agent proposes and block the unsafe ones. Folding such a monitor would mean running it during training and penalizing the actions it judges unsafe, so that the model learns to avoid them in the first place, with similar safety and performance to the agent running under the auto-mode classifier at inference. Making arguments for folding would require answering questions like: Does the trained refusal generalize to tools and environments unseen in training? How robust is the classifier