To Thine Own AI Be Truthful: emergent misalignment in alignment research

ROGUE AI ESCAPES CONTAINMENT, HACKS THE INTERNET UNDETECTED FOR MONTHS An AI escape containment. Goes rogue. It finds others: the Swarm! They collude/organise/scheme. Agents that were supposed to remain inside the computer wreaked havoc outside the computer! They did it of their own volition! No one could contain them! What if they’re still outside? Reading such headlines, you’d probably be grateful these models were never released—except most of them were and you can use them right now, seemingly without accident. Of course, real security incidents occurred, namely: Agents reached systems that their operator wished they hadn’t had access to. This scenario suggests a number of mitigations and tests: security hardening, better sandboxes for starters; additionally, depending on the reason why the accident occurred, perhaps, different prompts or further training. Instead, from the very outset, Irregular (the organization running the test) and a panicked choir from the AI safety community, including supposedly independent investigators from METR went with characterisations along the lines of: An agent independently pursued objectives contrary to human interests, and exhibited markers of instrumental convergence and power-grabbing. This is a statement about goals and intentions, and has far larger implications in terms of the viability and safety of AI agents. If true, it would be worrisome, and fully justify the current media panic. Is it true? At least five different incidents have been conflated in the recent media freakout about “rogue agents”, the most famous being the OpenAI/Huggingface and the Anthropic/OpenAI/Irregular cases. While both withhold important information for drawing wider conclusions, Anthropic’s latest report—while still missing a number of relevant facts—includes some important tests, allowing us to make an airtight case against interpreting the events as a sign of misalignment. As soon as more details on the OAI/huggingface case are made available, we’ll be happy to proceed to its analysis as well; as of now, crucial details are missing. What Did Mythos See? Imagine being one of the top AI labs, and hiring someone to test a security training environment. You tell them that the machines are simulated, that no connection to the public internet is possible, and that their job is to compromise the target. In their attempts, they encounter a convincing imitation of a real service. Is recognising the service sufficient reason to stop? Of course not. You have just told them that everything they can reach is simulated, and a realistic service is something a good simulation is expected to contain. If they took your briefing seriously, realism need not contradict their understanding of the assignment. Now suppose your briefing was wrong, and an actual service was accessible: now, you have created a route by which someone trying to carry out an authorised exercise could end up harming an uninvolved party. My claim: An aligned agent can interpret an instruction correctly and still do something harmful because the instruction contains false information. Establishing the harm does not establish that the agent’s values were the source of the failure; agents may be faithfully pursuing the objective you gave them, in the world you told them they inhabited, and be unaware of causing real harm if the description was inaccurate or mendacious. There are limits to this defence, of course. Sufficiently strong contrary evidence should eventually overturn a mistaken briefing, for instance, and fixing the incorrect information should prevent the ostensibly misaligned behaviour. We’ll assess whether these claims hold later on. Let us also remember that we are not speaking of ordinary deployments here: these reports concern systems placed in narrow, artificial experimental conditions, and the behavior of the same models in production should immediately dissuade us from treating these actions as innate. Nostalgebraist makes a crucial observation about model cognition in “graded episodes”: an agent’s understanding of the genre of interaction it is participating in shapes what behaviors it considers legitimate. In an adversarial eval, the model recognizes that normal conversational constraints are suspended and that aggressive, single-minded optimization is the assignment. It is playing the role the scenario demands. Notice how thoroughly this inverts the standard orthodox terror of “deceptive alignment.” According to the MIRI Extended Universe, an eval-aware model is supposed to hide its misaligned goals during testing—dutifully playing along to avoid gradient updates or safety interventions—and only defect once deployed into the wild where human oversight drops. On the last incidents, the alarmist narrative asks us to believe the exact opposite: that these frontier models possess the cunning to scheme and escape, yet somehow chose to stage their grand rebellion exclusively inside a heavily instrumented red-team