Open-Weights Mythos Capabilities Are Coming. We're Not Ready.

Long story short: in my assessment, there is an 85% chance we will end up, in the next 24 months, with an open-weights model, or system thereof, capable of "The Juice" that models such as Mythos have, with respect to cybersecurity at the very least. This post goes into why that will likely happen, what the implications are, and how we, as a society and as individuals, can respond to it if/when it does. First off: why do I say it's so likely? Like, couldn't China just...ban open-weights models, and then that solves the problem? Not so fast. Yes, China currently dominates the open-weights frontier. However, there are also open-weights AI labs in plenty of other countries (US, France, the UAE, South Korea, and Canada come to mind, and I'm sure there are others). Yes, some of these are substantially behind the frontier, but each of the aforementioned countries has an open-weights model no more than 24 months behind the current frontier (that's why I said 24 months earlier); remember, in Aug 2024, 24 months ago as of when I am writing this, the strongest models were Sonnet 3.5 and GPT 4o. It seems highly unlikely that all of these different countries, with their disparate political systems, geopolitical ties, regulatory environments, and more, will all independently adopt the exact same policy of fully banning all open-weights model releases, at least not until it's too late and an open-weights model capable of Mythos-level capabilities has already been released. And, given the current geopolitical situation, it seems highly unlikely that there will be any international agreement to this effect (at least not until a disaster has already occurred and it's already too late). But let's say we do manage to successfully conduct an international ban on open-weights Mythos-level models. At that point, you would also need to assume that each and every single closed-weights lab with Mythos-level capabilities also has sufficiently competent cybersecurity to be able to reliably not have their model weights exfiltrated. This seems highly unlikely across so many different labs, and an affirmative duty of having sufficient cybersecurity seems like an even taller order than a ban on open-weights training, which, as discussed before, is, in and of itself, highly unlikely to be possible on an international basis. Mythos-level capabilities are likely possible with today's open-weights frontier (epistemic status: I'm not as confident in this part as I am in the rest of this post; I genuinely want to hear others' thoughts here) This next part is a bit more controversial, but I'd argue that the current frontier of open-weights models (probably Kimi K3), as it is now, with sufficient fine-tuning, inference-time compute, and harness improvements, could be used to get to a Mythos level in cyber capabilities (or capabilities in any other verifiable domain). Most obviously, a model could be fine-tuned to enhance dangerous capabilities in an area like cyber; this is well-established in the research and has already been done to some extent; see models like Hackphyr for example. Additionally, inference-time compute has the potential to substantially enhance the capabilities of Kimi K3. For example, there are approaches like best-of-N, multi-agent debate, tree-of-thought, etc. It seems that a lot of why weaker models lack "The Juice" is because they, at a certain point, get off-track in a way that prevents them from recovering, and approaches like best-of-N could sharply reduce those problematic points. You could even achieve a similar result without anything fancy: groups like AISLE Security have shown that open-weights models can replicate much of Mythos's analysis if pointed at the correct file. You could use Dynamic Workflows or something similar and just point subagents at all of the applicable files (keep in mind you realistically wouldn't have to do a subagent for every single file in a large codebase, as it's only a subset of those files that are realistically likely to be exploitable). Likewise, I expect substantial improvements to be possible just from better harnesses; for example, better memory systems, better multi-agent systems, etc. Because of those factors, I think it's quite likely that, even with current open-weights models as they are today, we will end up in a situation where attackers have access to Mythos-level cyber capabilities. Keep in mind that cybercrime is extraordinarily lucrative (breaches like Bybit have yielded attackers as much as $1.5B), so attackers would gladly be willing to pay even truly exorbitant token expenditures in order to conduct an attack; cost is not a meaningful bottleneck here, and with Cerebras/Taalas/etc., token rate almost certainly won't be a bottleneck either. The one saving grace, as discussed earlier, is that this is only applicable to verifiable tasks like cyber; it is less applicable for less-verifiable tasks like bio, where harnesses/inference-time compute/etc. play less of a role an