Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study

Abstract:Continual Learning Bench scores learning as short-horizon gain versus a reset baseline and finds naive full-context ICL strongest among the memories it tested. We treat ICL as one learning system and score it on a longer shared-world schedule. Steady-state learning is the gap versus baseline on a pre-set late window (last 40 of 174 BIRD-SQL formula-1 questions). We split the score into exploration efficiency (SQL probes), task reward (hits), and delivery cost (API dollars and context size). On gpt-5.6-luna, late probes fall from 4.6-5.6 to 0.95 while hits rise only modestly and ICL context grows to about 95k tokens with cost roughly doubling. Short-horizon gain understates the late probe saving and misses the cost inversion, so we find that unbounded ICL is a poor candidate for the learning mechanism.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2609.31640 [cs.LG]

(or arXiv:2609.31640v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.31640 Focus to learn more arXiv-issued DOI via DataCite

f1-luna-r1-icl-vs-baseline.html

harbor-hub-urls.txt

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.