Updating the Epoch Capabilities Index methodology to use a Bayesian model

We have updated the modeling behind the Epoch Capabilities Index (ECI), which measures and tracks AI model capability using benchmark results. The primary motivation is to provide better-justified uncertainty estimates and to let us calculate how likely one model is to be more capable than another. The new methodology is documented on our website; this post explains how it differs from the previous frequentist approach and what motivates the changes. We first review the basic setup of the ECI and the previous method, then detail the changes in the Bayesian model and present its results. The appendices cover alternative formulations and the evaluation we used to validate our final modeling choices. The ECI is based on Item Response Theory (IRT), a framework originally developed for educational testing that enables comparisons between students even when they took different exams with varying difficulty. Similarly, we have different sets of benchmark scores for AI models that we want to compare. While classical IRT models the binary chance of answering a single question correctly, we adapt it to continuous outcomes by treating a model’s benchmark score as the expected share of questions it gets right. We only use benchmarks that are scored between 0…