What is the Alignment Community Thinking?

Compassion Aligned Machine Learning (CaML) is conducting research and deploying benchmarks with the aim of implementing broad compassion for all sentient beings into AI systems. In so doing, we’re attempting to integrate ideas from across the broad field of alignment research, including those focussed on specific technical questions, and those considering more strategic questions around who or what AI should be aligned with. To do this, it's important for us to understand how the alignment community is conceptualizing and prioritizing different ideas that CaML cares about. In this spirit, we recently conducted a survey on controversial questions in alignment, to encourage more discussion on topics important to CaML, and roughly gauge where both core researchers and wider community members stand. Questions ranged from the value of benchmarking as eval awareness increases, to the implications of AGI for animals, to thoughts about AI s-risks. We’ve shared some interesting findings from this survey below. Our intent with this survey was not to conduct an exhaustive analysis of the community’s position on every issue. This was an initial probe primarily designed to generate discussion and feedback on issues important to CaML. That said, we’d love to see research more accurately tracking core areas of (dis)agreement, consensus and controversy within alignment research. If we know where there is uncertainty, we can better understand where our research should focus next. What We Did Our survey consisted of two stages. In the first, we shared 14 polls with a panel of 14 researchers, where they privately responded to statements on an 11-point agree-disagree scale. They could also share the reasoning behind their answers. We received responses from the following 14 researchers: Scott Alexander (Astral Codex Ten) Cameron Berg (Reciprocal Research) Oscar Horta (Animal Ethics) Jeff Sebo (NYU CMEP) Chris Pang (NVIDIA) Tobias Baumann (CRS) Michael Dickens (Independent) Dawn Drescher (Impartial Priorities) Tristan Katz (Rethink Priorities) Max Taylor (ACE) David Manheim (ALTER) Cameron Holmes (Independent) Javier Ferrando (Amazon) Raphael Sarfati (Goodfire) Please note that respondents took these polls in an individual capacity, and responses do not represent the views of their employers In the second stage, we released the survey totalling 15 polls (with one additional poll to the panel survey) to the broader community. Respondents answered via an EA Forum post (where all the polls are still viewable), and could publicly share their reasoning for different responses. After feedback from the expert panel, we included clarifications alongside the polls. We received a total of 1444 votes over all 15 polls, and over 100 comments providing further reasoning. Once the survey was completed, we analysed two primary variables across the 15 questions: the lean (whether respondents on average voted agree or disagree), and standard deviation (SD) to see the spread of responses on a given question. Lean is measured from -5 meaning strongly disagree to +5 meaning strongly agree (with 0 being neither agree or disagree). What We Found Collective lean per statement, sorted by community lean. Grey connectors join the community and panel means for the same statement; rows where the dots sit on opposite sides of the dashed line are statements on which the two samples lean in opposite directions. The right-hand column gives the standard deviation of responses for the community (C) and panel (P). *Asked to the community only. Most People see the s-risk field as underperforming In two polls, we asked explicitly about the value of S-risk work: Poll six: At the margin, S-risk work in AI is more important than X-risk work. Poll eight: We are making good progress in the AI S-risk space, and research is on track. On both of these questions, the community leant towards disagreeing with the statement (lean of -1.2 on poll six, and -2.0 on poll eight). Only 3 of 73 respondents agreed that s-risk progress was on track and not one strongly agreed (lean of more than 1.5). Poll six saw the biggest distance between the community and the expert panel, with the panel leaning agree (+1.2) that s-risk work was more valuable at the margin. The panel was somewhat split on this though: this poll had the highest standard deviation (3.2) of the set. This may have been somewhat representative of the different research interests of the panel, with some explicitly more focussed on s-risks than x-risks or vice versa. Nevertheless, both the panel and the community disagreed that s-risk work was on track. Some text responses gave more detail as to why: ‘It’s notoriously difficult to say due to lack of clear feedback loops’ ‘Depends a lot on how good progress is defined’ ‘The space seems highly contested, deeply illegible, few viable research directions are immediately present’ Regardless of how respondents prioritise s-risks, there was at least agreement that, as i