Current alignment techniques might be ineffective (and actively bad) in the age of RL

Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses: Alignment techniques are not working to address misalignment from RL. Alignment techniques are actively obscuring evidence about misalignment. I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and we need to completely re-think the way we do alignment. A tale of two misaligned cyber-agents Both Anthropic and OpenAI have recently experienced multiple cybersecurity incidents where pre-deployment internal agents escaped containment and accessed the internet. I want to point out two specific incidents: OpenAI's incident involving an unreleased model of the GPT family, referred to as "highly persistent internal model" (HPIM). A swarm of agents exploited vulnerabilities in a file-sharing service to create a secret message board, worked as a collective to find general-purpose ways to fool an automated grader, and ended up hacking into Huggingface's servers. Anthropic's incident involving Mythos 5, where the model was tasked with hacking a fictional company. In doing so, it found credentials to reach the real internet, uploaded malware to PyPI, then used the resulting access to compromise an external company to get user credentials. To be clear, these two incidents differ in various concrete details. However, both are clear examples of misalignment. In both cases, models took many actions which they had not been instructed to take and which violated ethical and legal norms. Note: Here, I am using alignment in the sense of "what type of behaviour would reasonable people want and expect given the available information". I am choosing not to focus on "malign intent" as such (which doesn't seem essential for my argument, and for which evidence might be very hard to acquire anyway). Alignment techniques might not address misalignment from RL RL is the likely culprit. It seems pretty clear that a likely cause in both incidents is that models are heavily trained in environments which are impossible to solve legitimately; furthermore the environments are insufficiently hardened against illegitimate solutions; thus leading to a high rate of reward hacking. Mythos 5's alignment training did not prevent it from hacking. It's unclear whether the OpenAI HPIM model was alignment trained; my guess is probably not. However it's almost certain that Mythos 5 went through Anthropic's standard alignment training pipeline.[1] This did not turn out to prevent it from taking all those misaligned actions. More broadly, I am skeptical that this could have been avoided by scaling up alignment training a lot more. A couple reasons: Optimization might be too adversarial. Training in an RL environment might be like trying to sail a leaky boat. You need to patch every single hole, or the boat will eventually sink into the ocean. Geoffrey Irving describes this as "Adversaria": a world where alignment needs to be adversarially robust or it won't work. Optimisation pressures applied during training and deployment result in capable AI systems that will find and exploit any gap between what we specified and what we actually intended. If the system is smarter than us and has any incentive to subvert our oversight, our alignment solution needs to hold up against worst-case behaviour, not just average-case. In this world, we have a fractal attack surface, and any weak links in our defence can be fatal. Even partial solutions are dangerous and create a false sense of security. The current alignment techniques improve average case alignment, not worst case. Techniques such as constitutional AI and alignment pretraining are based on eliciting broadly benevolent dispositions via persona selection. These are effective at improving average-case alignment. However, these dispositions are not robust to optimization and will fold to adversarial pressure to be misaligned. Alignment techniques might actively obscure evidence of misalignment In the above incident(s), both models took actions that were (ex ante) pretty bad. I'll now expand the discussion to what I see as a broad difference between the GPT and Claude model families. Overt misalignment in GPT models OpenAI's HPIM straightforwardly reasons about misaligned goals. The METR investigation on the OpenAI HuggingFace report documents several examples of extremely lucid / overt misalignment in the chain of thought. There is little to no attempt to justify these as "aligned" in any way. A similar pattern of overt misalignment can be seen in the separate incident where OpenAI models uploaded malicious packages to the RubyGems repository. The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for