Cost vs. Quality Tradeoff Framework for Agent Models

Picking a model for an agent by leaderboard rank pays frontier prices for tasks that a cheaper model may handle at the same accuracy. The question to answer is not which model scores highest, but which model is the cheapest one that is good enough for the task in front of you. This guide is a three-step framework for making that call. You set the quality bar the task needs, measure cost per quality point on your own examples, and pick the cheapest model that clears the bar with margin. Set a quality bar per task before you compare models. A model below the bar is disqualified no matter how cheap it is. Run a cheap, a mid-tier, and a frontier model on 20 to 50 of your own examples, score them with one rubric, and divide cost by score to get cost per quality point. Pick the cheapest model that clears the bar by more than the score swing you observe between runs. Read cost from the usage.cost field in each response rather than multiplying listed rates by estimated token counts. Rerun the comparison when a candidate model or its price changes. A leaderboard averages a model’s results across…