aiqbench analysis
This benchmark demonstrates that specialized decision models like Jev can achieve near-frontier accuracy while delivering orders of magnitude improvements in latency and cost efficiency for classification tasks. The data suggests that organizations deploying high-volume intent routing systems should consider swapping general-purpose chat models for these smaller, purpose-built alternatives to optimize operational economics without sacrificing significant performance.
Is Jev as Accurate as Frontier Models at Classification?
Claude Opus 5 leads OpenRouter's classification task ranking by spend. We sent the same 3,080 Banking77 utterances to it and to Jev 1.13 through the Decisions API. Opus scored 84.4% to Jev's 81.0%, and Jev answered in 175 ms at $0.11 per thousand requests against 2.3 seconds and $2.42 for Opus.
Jev is TypeSafe’s System One decision model. Give it an app state object and a typed question, and it gives you back a typed answer, a confidence score, and the probability for each possible option (the Decisions API response type, not the chat endpoints). According to the TypeSafe pitch, you can replace a frontier chat model with a small judgment model like Jev and get significant cuts in both cost and latency, on classification tasks such as this one. How much accuracy are you sacrificing? We ran 3,080 Banking77 customer support utterances through both Jev 1.13 and Claude Opus 5, the model OpenRouter users spent the most on for classification in the Task spend section of its rankings page as of 22 September 2026. Each utterance gets sorted into one of 77 banking intents, and both models got the same one-line description of each intent. The chart below puts Jev and Opus side by side, showing how they compare on accuracy, median latency,…
Excerpt shown under the publisher's RSS terms · 163 words · full text and figures are on the publisher's site
Related coverage
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · Oct 8, 2026
Evaluating Decision Models for Text Annotation in Computational Social Science
arXiv · Sep 22, 2026
Cost vs. Quality Tradeoff Framework for Agent Models
OpenRouter · Oct 1, 2026
Scaling document classification to 100k+ labels
Databricks · Jul 20, 2026