Open-sourcing evals for open-weight agents

With new model releases like Fable and Gpt-5.6 Sol, it's becoming increasingly clear that closed-weight models are becoming a very expensive commodity, leaving us at the mercy of frontier labs to tell us what to use and how much. https://x.com/levelsio/status/2075530242186166519 Most coding-agent subscription plans lack pricing transparency, and the credit limits could have different token values depending on the day. Even the most pro-AI-usage, token-maxxing companies are now finding that their options are either to default to open-weight models or simply to make their own harness for efficiency. At Cline, we noticed an unusual rise in the token usage of open-weight models in the last few months for obvious reasons, such as price and quality improvements that make open-weight models on par with or better than closed-weight models. After stack-ranking our usage, we saw that the five most-used models in Cline right now are all open-weight models. So we started digging into how Cline performs on open-weight models. Most Cline users are spoiled SOTA token-maxxers who burn billions of tokens on the latest frontier models. But all of us are experimenting more with open-weight models, learning how to work with them and how to improve them. Our new SDK lets…