Rerunning AI safety papers on every frontier release would be pretty easy and valuable

tl;dr: Some important AI safety research is never rerun on the newest models. There are probably cases where this would be valuable and a single well-positioned researcher could likely do this with sufficient funding. This summer, Second Look Research (SLR) is running a summer fellowship dedicated to empirical replications of AI safety research. Many of our most interesting results so far came from replicating previous results on newer or more capable models.  For example, it is perhaps useful to know that Google’s CoT monitorability experiments continue to hold for models like GPT-5.5, which are qualitatively more capable than the models originally tested. Likewise, continuing to track Ryan Greenblatt’s filler token results on more capable models gives a fuzzy signal indicating how much newer models can use innocuous tokens to hide additional reasoning in a forward pass. These kinds of experiments do not lose value over time! It’s important to track whether safety-relevant model properties still hold in new model releases and to be aware of any changes. It can sometimes be difficult to rerun results on newer models because codebases can be incomplete, have parameters that differ from the original paper, or may not be open source to begin with. We ran into all of these problems throughout the fellowship when trying to replicate experiments. But because SLR has already gone through the effort to replicate the original results of several influential AI safety papers, we know that we have codebases that are complete, produce numbers that won’t surprise us, and could be updated for new models. For papers that already have a replication, adding a new model is a single terminal command away, or at the most a single Claude prompt away. What could this actually look like? We imagine that when a new model comes out, a competent undergrad we trust can run several experiments we select overnight and review & verify the results. Depending on the paper, this costs somewhere between $0 and $5,000.[1] If the undergrad surfaces something sufficiently important or a step change, we could immediately communicate this result to stakeholders. If they don’t find anything interesting, we can simply post a result to our website saying we didn’t find anything unexpected. All research communication will be read over by a more experienced member of the team before being pushed out. After these experiments are completed, we can update a tracker of current trends we see in the different experiments we are tracking. This may surface subtle safety trends that nobody would notice otherwise. Does this actually provide value? Or a related question: Aren’t the most important evaluations and model properties already covered in model system cards? It's possible that a lot of research goes stale because it's not that interesting and that the field is pretty good at deciding which properties are relevant enough to keep testing. Additionally, corrective feedback loops already exist: for example, if a model card doesn’t evaluate something it should, people can complain and the company can adjust that in their next model card. Within what would traditionally be considered evaluations (capability & risk), the most useful ones do get run.  However, our replications cover very different questions than typical evals run by organizations (e.g. METR, Apollo, Palisade) and frontier labs. We would run full experiments from safety papers, some of which include an eval phase, but wouldn’t be counterfactually performed by others doing alignment evals. While Anthropic’s model cards contain safety-relevant information, there are some tests that are consistently left out even though they aren’t that expensive or difficult. OpenAI, GDM, and xAI can have very limited system cards. Overall, we notice that work that is less evals-shaped (e.g., internal state control, peer preservation, or filler token evals) is less likely to be included in a system card or run by an external evaluator and we think we can cover some of this gap. As a sample, these are the kinds of results we would consider targeting: Teaching Claude Why midtraining experiments on new open source models (replication coming out in the following weeks) CoT hint following experiments (replication here) Single-forward pass filler token evals (replication here and more results coming soon) Internal state control on new open source models (replication here) Hidden role games to measure adversarial reasoning under uncertainty (original post here, replication coming soon) Subliminal learning (original paper here). We expect subliminal learning to replicate on new open source models but quantitative trends may be interesting.[2] Logistical challenges with continuing to update AI safety research with new models What if people don’t want to do this? The most obvious reason people don't do this is that it’s not incentivized, unlikely to turn into a full publication, and therefore top talent isn’t interested