Metrics of Agent Ability

Notes are unpolished research updates directly from METR researchers, and have gone through less review than our research posts. They do not necessarily reflect the views of METR as a whole. Expenditure can be interpreted as money spent, or tokens, or time required. I will not give a deep discussion of how to measure the score curves, $s_H$ and $s_A$. There are many practical difficulties involved in this. I will generally address only scores on a single task; applications to collections of tasks (benchmark scores) can be a bit more complicated. I will mostly not discuss other desirable properties of metrics - e.g. whether it is understandable, whether it is generalizable. If score curves are very concave (meaning score is invariant to expenditure) then just reporting the raw score is fine, given some reasonable amount of expenditure. If score curves are very concave (meaning score is invariant to expenditure) then just reporting the raw score is fine, given some reasonable amount of expenditure. If score curves show high returns to expenditure then we need a metric that takes into account test-time scaling. If score curves show high returns to expenditure then we need a metric that takes into account test-time…