aiqbench analysis
This article examines the capabilities and safety implications of advanced reasoning models like GPT-6 (Astra) when configured for autonomous agent swarming. The author highlights that while these systems can effectively delegate tasks and manage sub-agents, their current "raw" nature requires significant token budgets for evaluation, making thorough interrogation of swarm behavior a critical priority for AI safety researchers.
[Linkpost] Looking into the Swarm's Eye
I'm Florian Brand is currently working as Research Engineer at Prime Intellect. Currently, my research focuses on applying and evaluating LLMs in various domains. I am also an editor at Interconnects, focusing on open models. There is, however, a big gap between open models in a suitable harness and GPT-6 (Astra), the first model trained very deliberately to be a capable RLM. Astra is currently held back by its native harness, Codex, and its default prompts. When elicited correctly, it is a sight to behold: It can delegate work effectively, manage its subagents, spawn (sub-)subagents on its own when appropriate, and let all of them communicate with and about each other. It is also very raw as a model, making mistakes and being close to an alien mind. Similar to o1-preview, these issues will be worked out over time and the models will become more reliable, but this makes the current generation of models all the more exciting. As mentioned in the post, we're still early days in exploring swarm behavior. It is currently expensive to do so. Currently, it seems like you need token budgets in the $10-100ks to sufficiently explore them. Interrogating the behavior of swarms is urgent for AI safety. If you're someone with the budget and competency to design evals and interrogate swarm behavior more thoroughly, please do so! Discuss
I'm Florian Brand is currently working as Research Engineer at Prime Intellect. Currently, my research focuses on applying and evaluating LLMs in various domains. I am also an editor at Interconnects, focusing on open models.
I'm Florian Brand is currently working as Research Engineer at Prime Intellect. Currently, my research focuses on applying and evaluating LLMs in various domains. I am also an editor at Interconnects, focusing on open models.
There is, however, a big gap between open models in a suitable harness and GPT-6 (Astra), the first model trained very deliberately to be a capable RLM. Astra is currently held back by its native harness, Codex, and its default prompts. When elicited correctly, it is a sight to behold: It can delegate work effectively, manage its subagents, spawn (sub-)subagents on its own when appropriate, and let all of them communicate with and about each other. It is also very raw as a model, making mistakes and being close to an alien mind. Similar to o1-preview, these issues will be worked out over time and the models will become more reliable, but this makes the current generation of models all the more exciting.
There is, however, a big gap between open models in a suitable harness and GPT-6 (Astra), the first model trained very deliberately to be a capable RLM. Astra is currently held back by its native harness, Codex, and its default prompts. When elicited correctly, it is a sight to behold: It can delegate work effectively, manage its subagents, spawn (sub-)subagents on its own when appropriate, and let all of them communicate with and about each other. It is also very raw as a model, making mistakes and being close to an alien mind. Similar to o1-preview, these issues will be worked out over time and the models will become more reliable, but this makes the current generation of models all the more exciting.
As mentioned in the post, we're still early days in exploring swarm behavior. It is currently expensive to do so. Currently, it seems like you need token budgets in the $10-100ks to sufficiently explore them.
Interrogating the behavior of swarms is urgent for AI safety. If you're someone with the budget and competency to design evals and interrogate swarm behavior more thoroughly, please do so!
Full text under the publisher's open licence · 381 words
