aiqbench analysis
JevRank turns LLM-based decision models into rerankers that run through a user's own LiteLLM endpoint. Its measured SciFact gain (nDCG@10 from 0.686 with BM25 to 0.768, at an estimated $0.575 per 1,000 queries) is the most useful number, but it comes from one 300-query benchmark. The cross-vendor comparison relies on reference results that the authors say were not verified to match their setup, and costs exclude gateway overhead, so treat the price and quality claims as directional.
JevRank: decision models through LiteLLM’s rerank API
Preview JevRank for low-cost reranking with Perplexity and TypeSafe decision models. Explore SciFact results, reproducible data and LiteLLM setup.
Turn Perplexity and TypeSafe decision models into rerankers. Keep your query, your documents and your LiteLLM endpoint. JevRank builds on the decision-model reranking research of Jakob Gruber at Swietelsky AG. In a SciFact reference benchmark, Perplexity Flash and Cohere Rerank 4 Pro both scored 0.775 nDCG@10. Their projected costs were $73 versus $2,500 per million queries, about 34× lower for Flash. Perplexity Decider Flash v1.2 (JevRank)Lowest reference costMultiple choice Reference nDCG at 10: Quality ↑0.775 Perplexity Decider v1.2 (JevRank)Multiple choice Reference nDCG at 10: Quality ↑0.777 TypeSafe Jev 1.14 (JevRank)Per document Reference nDCG at 10: Quality ↑0.776 nDCG at 10: Quality ↑0.775 SciFact · 300 queries · 20 candidates · October 2026. Reference results via OpenRouter. Flash costs $73 versus Cohere Pro’s $2,500 per million queries, about 34× lower. These are reference results, not our LiteLLM run. Costs exclude gateway overhead; Jev 1.14…
Excerpt shown under the publisher's RSS terms · 143 words · full text and figures are on the publisher's site
