Learnings from starting an AI safety research team

This post’s goal is to distill our takeaways from building a new research team over the past four months. We describe some context about our team, how it came about, and then describe the lessons learned. Since AI safety is becoming more and more entrepreneurial, we hope this is helpful for others trying to do the same. 1. The team We're a new alignment research team within Arcadia Impact, based in London. We’re a team of 8, working closely with members of the UK AISI alignment team. We currently have three main projects: Understanding model motivations. This currently looks like: Trying to generate documents which fully describe a model’s behaviour (given just its behaviour). Producing an open analysis of alignment training techniques and ways this training could go wrong. Doing scalable oversight for alignment. This includes validating debate protocols in practice and then trying to apply them to fuzzy alignment-relevant tasks. Building pipelines for doing automated alignment research. We're also hiring for two roles! More on this at the bottom. 2. Context about how the team came about The rest of this post is written from the perspective of Andrew Draganov (research lead & current programme manager on the team) and Erin Robertson (co-director of Arcadia). In short, Arcadia Impact had been collaborating with AISI already, through LASR Labs and ASET. Our alignment team started by applying for the AISI alignment project funding, saying that we would hire a team of researchers to collaborate with their alignment team. Andrew was taking part in LASR at the time and was brought in to help with the application. His remit then widened as the number of things to do kept growing. Once our AISI funding was approved we began the process of hiring researchers, and also applied to Coefficient Giving for additional compute funding. A bit about Andrew, since it bears on how replicable this is. In his words:  I have a PhD in computer science/machine learning and was working as a postdoc in ML before doing LASR. This means I've spent a number of years talking shop about AI research, though not as many on AI safety specifically. I'm not very well-known in the AI safety community! I only have one first-author AI safety paper (which was reasonably well-received but nothing crazy). I mention this because "you need to be an established name to lead a research team" is a reasonable thing to assume, but it wasn't really true here. For anyone reading this post as a template, here are some things which may be specific to our situation and might not generalise cleanly: We were immediately hiring 7 researchers to get started at the same time! This is highly unusual and probably never how this otherwise happens. Arcadia was already an established non-profit. We therefore already had visa sponsorship processes, office space, hiring systems, etc. There are fiscal sponsors which can do these tasks if you want to avoid figuring out the overhead yourself. The Alignment Project, run by AISI, was our initial funder. This is a non-standard funder for many reasons, including that Arcadia already had a working relationship with AISI writ large. If you're aiming to first get funded by, say, Coefficient Giving then the dynamics may be different. Having run LASR, we know a lot of people in the ecosystem quite well. This made hiring easier (and, indeed, over half of the team are LASR alumni). We're doing technical AI safety; not governance, fieldbuilding, etc. 3. Lessons learned Given the above context, here is advice which we hope is immediately actionable by people looking to start AI safety orgs. 3.1 Hiring [Written from Andrew’s perspective] I feel like our hiring went very well and I’m really excited about the team. But also I wasted a lot of time chasing leads that were varying amounts of useful.  For one thing, everyone wants to measure 'crackedness' but it’s unclear how to do it. On that axis, the two highest-signal parts of our process were the work test and the references; if we'd relied on only those two, I think we'd have assessed raw research ability roughly as well as we did. The interviews were helpful in addition to that, but mostly to vibecheck for fit rather than to gauge ability. For the work test, we paid 50 applicants ~$200 each to make a research proposal. We gave them 4 hours to do this, and the deliverable was just a pdf. We then graded them anonymously. This feels in line with what the work actually looks like in the age of Claude code. We’re happy to share the work test and grading template we used if someone is interested. Here are a few additional thoughts: The various AI-safety talent scouts are extremely useful when it comes to hiring. This includes research fellowship research managers, people at BlueDot, people at 80K, etc. There’s just so much talent across the top fellowships. Our team ended up with 4 LASR alums, 1 MATS, 1 Astra, 1 Anthropic Fellow. Most of these fellowships now have extension programmes,