analiza aiqbench
This study demonstrates that behavioral cloning can effectively address the "last mile" bottleneck in scientific data annotation by modeling expert strategies like exploration and error correction. Key findings indicate that hierarchical skill emergence and multi-task pretraining significantly improve data efficiency and enable robust fine-tuning across diverse annotation tasks. By revealing shared internal representations for mistakes, the work provides a foundational framework for scaling automated verification in complex scientific workflows.
A Systematic Study of Behavioral Cloning for Scientific Data Annotation
arXiv:2606.07568v1 Announce Type: cross Abstract: Scientific data annotation, such as tracking animals in video or proofreading neural reconstructions, remains bottlenecked by the "last mile" problem: even with strong automation, verification and correction consume substantial human effort. Standard approaches train models to directly predict annotations, discarding the rich supervision in how experts navigate, click, verify, and correct. We introduce a framework for studying behavioral cloning on scientific annotation: 9 synthetic tasks paired with synthetic annotations that simulate realistic human strategies including exploration, mistake correction, and strategic decision-making. Our experiments reveal several findings. First, skills emerge hierarchically: models learn GUI mechanics before task-critical decisions, and commit fewer mistakes than the training data while retaining the ability to correct errors when they occur. Second, scaling models on multi-task behavioral cloning shows that larger models are more data efficient within our scale range. Third, multi-task pretraining enables efficient fine-tuning to new tasks, while training from scratch fails entirely. Fourth, linear probes reveal that models internally represent latent variables of the annotation process such as task phase and data position; interestingly, we find a shared mistake representation that generalizes across different annotation tasks. Overall, our framework establishes systematic benchmarks and identifies key bottlenecks, providing a foundation for scaling behavioral cloning to real-world scientific data annotation.
Abstract:Scientific data annotation, such as tracking animals in video or proofreading neural reconstructions, remains bottlenecked by the "last mile" problem: even with strong automation, verification and correction consume substantial human effort. Standard approaches train models to directly predict annotations, discarding the rich supervision in how experts navigate, click, verify, and correct. We introduce a framework for studying behavioral cloning on scientific annotation: 9 synthetic tasks paired with synthetic annotations that simulate realistic human strategies including exploration, mistake correction, and strategic decision-making. Our experiments reveal several findings. First, skills emerge hierarchically: models learn GUI mechanics before task-critical decisions, and commit fewer mistakes than the training data while retaining the ability to correct errors when they occur. Second, scaling models on multi-task behavioral cloning shows that larger models are more data efficient within our scale range. Third, multi-task pretraining enables efficient fine-tuning to new tasks, while training from scratch fails entirely. Fourth, linear probes reveal that models internally represent latent variables of the annotation process such as task phase and data position; interestingly, we find a shared mistake representation that generalizes across different annotation tasks. Overall, our framework establishes systematic benchmarks and identifies key bottlenecks, providing a foundation for scaling behavioral cloning to real-world scientific data annotation.
Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Data Analysis, Statistics and Probability (physics.data-an)
arXiv:2606.07568 [cs.HC]
(or arXiv:2606.07568v1 [cs.HC] for this version)
Text integral sub licența deschisă a publicației · 234 cuvinte
Subiecte conexe
Revisiting the Systematicity in Negation in the Era of In-Context Learning
arXiv · 17 iun. 2026
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
arXiv · 22 sept. 2026
A Bounded Evaluation Protocol for Representational Systematicity
arXiv · 3 nov. 2026
Sweet Talkers: How Query Formulation Shapes Sycophancy in Romantic Relationship Advice
arXiv · 15 sept. 2026