aiqbench takeAI Products

This is a product update for a specialized model used as an evaluator, not a new model architecture or benchmark result. The article frames it as a trade-off: faster and cheaper than LLM-as-a-judge, with more flexibility than code-based checks. Readers building agent evaluation pipelines should weigh those claims against their own tests of judge consistency and structured-output reliability, since the article itself notes those are known weak points.

Source coveringLangChain Blog

Jev is now available in LangSmith Evals

LangChain BlogSeptember 29, 20268 min read

Use Jev as a judge for LangSmith evals to evaluate agent traces with faster, cheaper structured feedback across production runs, datasets, and regression tests.

Jev is now available as a judge for evaluations in LangSmith. Jev gives teams a fast, low-cost way to evaluate open-ended agent behavior and turn the results into structured feedback they can track in LangSmith. Below, we explain why a System One model like Jev is useful for agent evals, share what we found when we tested it, and walk through setting up a Jev-as-a-judge evaluator for online evals. Try Jev-as-a-judge in LangSmith today by visiting the Evaluators tab in any tracing project. Back in 2023 when we first started building agents (which we mostly called LLM apps at the time), the primary approach to evals was code-based. Later that year, researchers introduced LLM-as-a-judge, and since then, code-based and LLM-as-a-judge have been the two main ways to evaluate agents. Code-based evaluators check for specific, deterministic conditions: Did the agent call a tool? Does the output match a pattern? Is a field present? That's fast and reliable, but…

Excerpt shown under the publisher's RSS terms · 157 words · full text and figures are on the publisher's site

Read at LangChain Blog ↗Ask about this article