Sari la conținut
aiqbenchInformații despre AI & calcul cuantic
ȘtiriAnalizeBenchmark-uriSkill-uriÎntreabă AIÎnvațăDespre
...Se încarcă sesiunea
ȘtiriAnalizeBenchmark-uriSkill-uriÎntreabă AIÎnvațăDespre

Română · beta. Traducerea a fost făcută automat și este încă în curs de revizuire. Citește pagina în engleză

Înapoi la listă
perspectiva aiqbenchSiguranță AI

analiza aiqbench

The article proposes a strategic framework for automating AI safety research, distinguishing between empirical experiments, conceptual theory, and direct system alignment. It argues that while automated agents are well-suited to run empirical alignment studies, this specific area may not be the highest priority for acceleration compared to developing better specifications for what we aim to measure.
1
Sursă care acoperăAI Alignment Forum

How I'm thinking about different types of automated alignment

AI Alignment Forum·11 noiembrie 2026·13 min de citit·Articolul și perspectiva sunt în engleză.

De la AI Alignment Forum

Text integral

I used to think of "automating alignment" as a natural category. But when I tried to think through the strategic implications, I realised that it depends a lot on the type of work. I'm now thinking of it in three buckets:

In the rest of this post I'll walk through my current takes on these in terms what’s needed for automation, how good or bad they look, what our priorities might be (to the extent we're interested in automating alignment[1]), etc.

This is the most obvious fit for the AI capabilities we’re getting at the moment. You want to know more about the relationship between some alignment-relevant variables? Great, an AI agent can design and run that experiment for you. You want to know what training methods help your alignment benchmark go up? In the not-too-distant future, AI agents will probably be able to do this too. When we have the automated capabilities researchers we seem on track for, this is exactly the kind of thing they’ll likely be able to do a lot of.

Note that I’m not counting the design of benchmarks/metrics/evals as part of this bucket. In current practice that’s tightly intertwined with empirical alignment research, and often the hardest part of it. But for automation purposes it clusters much more with conceptual work; it’s easy to imagine automating the rest of empirical research without it (as AI capabilities generally get automated); and I think its automation has different strategic upshots.

The capabilities that are needed for automating empirical alignment research very heavily overlap with the capabilities needed for automating AI capabilities research. Therefore, it doesn’t seem like a priority to try to accelerate this type of automation.

However, the prospect of getting a lot of automated empirical alignment work before we have good versions of other types of automated alignment work increases the priority of work which complements it. Significantly, that may mean better specification of the kind of numbers we would like measured/targeted.

What about automating the type of metric choice/construction that we’ve just said would be a good complement to automating empirical alignment work? Or more generally improving our understanding of how to think about alignment, what we’re even aiming for, etc.?

This is essentially theoretical research. Automating it is trickier than automating empirical work because of the lack of verifiable signal about what constitutes progress.

Accelerating the automation of conceptual research is somewhat fraught (feeding into some dangerous technologies), but seems helpful for a number of high-priority areas, including macrostrategy as well as conceptual alignment work. Overall it seems likely good to do, if we can.

As well as learning more about what it takes to make systems more aligned, sometimes we want to just make our actual systems as aligned as possible.

Automating direct alignment work could include a lot of activities which leverage scalable cognition for alignment-relevant purposes. This could include careful auditing of particular RL environments or training data, or direct monitoring/supervision of systems during training. RLAIF for alignment properties is a form of automated direct alignment work; Claude 3 Opus’s apparent self-alignment was another (albeit accidental in that case; but if we knew how to reproduce or generalize the behaviour perhaps it could happen deliberately).

I think that the limit of automating direct alignment work would be a form of system 2 learning, where updates are not made blindly towards a training objective, but reflectively by an AI system that was already sufficiently aligned (not necessarily having robustly good values already — just enough genuine care about being aligned, and honesty with itself, that its reflective updates tend to move it further in that direction), and wished to become more so — whatever the deep true version of aligned should mean[2].

Embarking on such a path could be fraught (because of difficulty in being confident that the starting conditions were in the right basin of attraction), but compared to other alignment approaches, it has the benefit that it might leverage the full strength of the AI’s cognition to help, so that its “self-improvement” in an alignment direction is a closer analogue to meditation than to self-performed neurosurgery.

Automated direct alignment techniques may be good subjects for empirical experimentsThis is not completely reliant on having good metrics for alignment; developing a science of what’s going on with different attempts at automated direct alignment could be useful even if we don’t yet know what “good” looks likeAutomated hill-climbing might get you to very strong direct alignment approaches — but this does rely on having good enough metricsi.e. we could plausibly be in a regime where we’re better at reasoning about and implementing training setups that in fact lead to aligned systems than we are at accurately measuring their alignment. In that case, optimising for our best alignment metrics might produce worse alignment outcomes than following our best-guess direct alignment process.

Automated hill-climbing might get you to very strong direct alignment approaches — but this does rely on having good enough metricsi.e. we could plausibly be in a regime where we’re better at reasoning about and implementing training setups that in fact lead to aligned systems than we are at accurately measuring their alignment. In that case, optimising for our best alignment metrics might produce worse alignment outcomes than following our best-guess direct alignment process.

i.e. we could plausibly be in a regime where we’re better at reasoning about and implementing training setups that in fact lead to aligned systems than we are at accurately measuring their alignment. In that case, optimising for our best alignment metrics might produce worse alignment outcomes than following our best-guess direct alignment process.

You could also in principle do a good enough job at direct alignment that the systems you’re building are entirely safeBut I think it will be hard to have very strong confidence in that before improving our automation of the conceptual side; and so each new generation of AI might pose a challenge — will your alignment techniques transfer, or will your luck run out?

But I think it will be hard to have very strong confidence in that before improving our automation of the conceptual side; and so each new generation of AI might pose a challenge — will your alignment techniques transfer, or will your luck run out?

There will likely be approaches that allow you to leverage automation of empirical research to get better automation of conceptual worke.g. Taking Talkie or another LLM with only vintage training data, and working out how to get it to perform well at rederiving more modern theoretical work

e.g. Taking Talkie or another LLM with only vintage training data, and working out how to get it to perform well at rederiving more modern theoretical work

How much loss-of-control risk is early?If there’s a bunch, then putting more energy into the “not losing” part of things makes senseIf the main misalignment risk comes when we’re getting systems that can perform well even on tasks without verifiable rewards (like long-term strategy to allow them to plot sophisticated power-seeking campaigns), then it makes sense to put more energy into accelerating the “winning” part, and so perhaps move it earlier in time so that it comes before some crucial challenge that we otherwise wouldn’t be able to faceNote that in this world, the win state and the biggest threats rely on capabilities that from our vantage point look quite similar — in that they require high-quality thinking about novel situations without verifiable rewards — so it’s quite plausible that even a modest acceleration could change the sequencing

How close might we be to decent automated conceptual alignment work, if we approach it relatively directly?If it’s not hideously out of reach, then it looks more like the kind of thing where human intervention to speed it up (by building benchmarks and scaffolds, and curating data) could make the differenceIf so, the full suite of activities mentioned above as precursors for conceptual alignment work looks helpfulIf automated conceptual research is too hard, maybe the direct interventions don’t actually do much — in this world, automated conceptual research falls out of stronger AI driving automated (empirical) research programs to work out how to automate conceptual researchSo trying to target conceptual alignment work directly might be obsoleted as a way to accelerate the “winning” moment, and it’s a higher priority to get things safe enough that we robustly don’t lose through the period where the automated capabilities research is speeding upI’d note that building metrics/benchmarks for conceptual research, that can be targeted by later automated empirical work, looks more robust to this concern than other forms of targeting automated conceptual research

My personal guesses on these uncertainties are that most of the loss-of-control risk comes later (after AI is superhuman even at tasks that don’t have verifiable rewards), and that it is possible to directly approach decent automated conceptual research. However, mostly I’m pretty uncertain — which argues in favour of hedging by investing some in all of the paths.

^I'm not saying "I think automated alignment is the best plan" (I currently don't; I'm more excited by epistemic and coordination uplift enabling robust coordination). But I think (1) it's a backstop to other plans and might work so is worth some investment, and (2) in any case it's helpful for assessing what we think of automated alignment to have a better sense of what it even means.

I'm not saying "I think automated alignment is the best plan" (I currently don't; I'm more excited by epistemic and coordination uplift enabling robust coordination). But I think (1) it's a backstop to other plans and might work so is worth some investment, and (2) in any case it's helpful for assessing what we think of automated alignment to have a better sense of what it even means.

^So the system would need something like the type of understanding that conceptual alignment research is aiming at. But conceptual research aims to understand things abstractly; the understanding which such a system had might be more akin to a lived experience and understanding. I say this without taking any stance on whether such AI systems would have conscious experience; it does rely on something like introspective ability, but that’s something we’ve seen (to some extent) in LLMs today.

So the system would need something like the type of understanding that conceptual alignment research is aiming at. But conceptual research aims to understand things abstractly; the understanding which such a system had might be more akin to a lived experience and understanding. I say this without taking any stance on whether such AI systems would have conscious experience; it does rely on something like introspective ability, but that’s something we’ve seen (to some extent) in LLMs today.

Text integral sub licența deschisă a publicației · 1790 cuvinte

Citește pe AI Alignment Forum ↗Întreabă despre acest articol

Subiecte conexe

Produse AI

Agentic AI vs AI Automation: What’s the Real Difference?

Analytics Vidhya · 21 iul. 2026

Cercetare AI

Automated Reinforcement Learning: An Overview

arXiv · 21 iul. 2026

AI & societate

Compensating the victims of automation is economically beneficial

LessWrong · 29 oct. 2026

Cercetare AI

AXRP Episode 50 - Eli Lifland on AI 2027

LessWrong · 3 aug. 2026

Article info

Source

AI Alignment Forum

Published

11 noiembrie 2026

Read time

13 min

Category

Siguranță AI
aiqbench
Informații despre AI & calcul cuantic© aiqbench. Toate drepturile rezervate.

Fluxuri:RSS·JSON Feed

LimbăEnglish·Românăbeta
ȘtiriAI & tehnologieCuanticAnalize
ExploreazăBenchmark-uriÎnvață
CompanieDespreÎncredereInformații legaleTermeni și condițiiPolitica de confidențialitateNotificări privind drepturile de autorȘtergerea datelorRaportează conținut

Perspectiva aiqbench, analizele, rezumatul zilnic și răspunsurile din Întreabă sunt scrise de modele AI din sursele afișate, fără revizuire editorială înainte de publicare. Pot conține erori — verifică sursa. Cum este folosit modelul →

  • Știri
  • Analize
  • Întreabă
  • Topuri
  • Cont