aiqbench analysis
This article evaluates whether frontier AI models can independently devise a machine learning innovation that matches a later human-developed one, using a benchmark called InnovationEval. Early results show little progress despite substantial GPU spend, which suggests end-to-end automated AI research remains far from demonstrated. For readers, the useful signal is the benchmark design, which tests genuine novelty against a held-out human result, rather than any claim that AI has automated research.
Can AI automate AI R&D yet?
AI developers aim to create an automated AI researcher. How close are they? Existing evidence shows that AI can perform software engineering tasks relevant to AI research, dataset creation,1 and other tasks, including open-ended optimization of defined metrics (autoresearch). But it remains unclear what full automation of AI R&D entails, given the wide range of activities involved. One obvious gap is the ability to conduct end-to-end research projects.2 Similar to recent work such as Crux Evals, ResearchGym, and RSI-Bench, we test AI on end-to-end research tasks. We present early results from InnovationEval, an evaluation where we measure AI’s ability to independently discover novel machine learning techniques comparable to those developed by human researchers. Recent frontier models make little progress on this task, despite running experiments using thousands of dollars’ worth of GPU time. We plan to expand and repeat this methodology, tracking AI’s progress towards automating AI research itself. InnovationEval tests whether AI can independently devise an ML innovation that matches the performance of a recent human-developed innovation the AI has not seen. This is similar to recently-proposed tests for scientific ideation: if AI were presented with humanity’s knowledge up to 1905, could it rediscover special relativity?3 We ask a…
Excerpt shown under the publisher's RSS terms · 200 words · full text and figures are on the publisher's site
.webp%3Fssl%3D1&w=320)
