Continual Learning Bench: measuring whether AI systems actually improve with experience

At our latest Snorkel AI Reading Group, Parth Asawa (UC Berkeley Sky Computing Lab) presented Continual Learning Bench, the first expert-validated benchmark designed to measure whether LLM-based systems genuinely improve through sequential experience. CL-Bench spans six domains, software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting, and every task sequence is built around a latent structure a stateful system can discover online but a stateless one cannot. The question it asks is simple and almost never asked: when a model reaches the tenth problem, did any of the first nine help? Lightly edited for readability. My name is Parth Asawa. I’m a PhD student at Berkeley working on continual learning, and I’m going to talk about a work we did a few months ago on evaluating it. We’re all at conferences and events like this one quite often, talking about what these models are doing. They’re learning, they’re really good at these various capabilities. But something you might notice is that we don’t actually talk about how much they learn. If you’re familiar with how we measure models today, you’ve seen the graphs. Every time a new model is released we get a wide variety…