ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

arXiv:2603.17216v2 Announce Type: replace Abstract: With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labor of machine learning (ML) research requires massive, process-level supervision. Existing static benchmarks omit critical intermediate steps such as debugging and incremental reasoning, and manual data collection is prohibitively expensive. To overcome this data bottleneck, we introduce ML-AutoResearch (ML-AR), a scalable pipeline for automatically generating synthetic, end-to-end ML research tasks. Each task defines a complete research cycle, including problem specification, dataset selection, baseline implementation, and iterative improvement. To ensure realism and executability, tasks are grounded in real-world datasets and refined via an automated self-debugging procedure without requiring human supervision. We construct a large-scale dataset of teacher trajectories on these synthetic tasks to train student agents via supervised fine-tuning. We evaluate the resulting agents across 3 diverse ML research benchmarks. Our comprehensive experiments across 2 distinct model families and 3 model sizes demonstrate that training on ML-AR trajectories yields consistent and significant capability gains. Fine-tuning improves the Aggregation Under the Performance (AUP) by up to 9\% and substantially boosts overall pass rates, highlighting robust out-of-domain generalization.

Abstract:With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labor of machine learning (ML) research requires massive, process-level supervision. Existing static benchmarks omit critical intermediate steps such as debugging and incremental reasoning, and manual data collection is prohibitively expensive. To overcome this data bottleneck, we introduce ML-AutoResearch (ML-AR), a scalable pipeline for automatically generating synthetic, end-to-end ML research tasks. Each task defines a complete research cycle, including problem specification, dataset selection, baseline implementation, and iterative improvement. To ensure realism and executability, tasks are grounded in real-world datasets and refined via an automated self-debugging procedure without requiring human supervision. We construct a large-scale dataset of teacher trajectories on these synthetic tasks to train student agents via supervised fine-tuning. We evaluate the resulting agents across 3 diverse ML research benchmarks. Our comprehensive experiments across 2 distinct model families and 3 model sizes demonstrate that training on ML-AR trajectories yields consistent and significant capability gains. Fine-tuning improves the Aggregation Under the Performance (AUP) by up to 9\% and substantially boosts overall pass rates, highlighting robust out-of-domain generalization.

Artificial Intelligence (cs.AI)

arXiv:2603.17216 [cs.AI]

(or arXiv:2603.17216v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2603.17216 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

Related coverage