Jev is now available in LangSmith Evals

Jev is now available as a judge for evaluations in LangSmith. Jev gives teams a fast, low-cost way to evaluate open-ended agent behavior and turn the results into structured feedback they can track in LangSmith. Below, we explain why a System One model like Jev is useful for agent evals, share what we found when we tested it, and walk through setting up a Jev-as-a-judge evaluator for online evals. Try Jev-as-a-judge in LangSmith today by visiting the Evaluators tab in any tracing project. Back in 2023 when we first started building agents (which we mostly called LLM apps at the time), the primary approach to evals was code-based. Later that year, researchers introduced LLM-as-a-judge, and since then, code-based and LLM-as-a-judge have been the two main ways to evaluate agents. Code-based evaluators check for specific, deterministic conditions: Did the agent call a tool? Does the output match a pattern? Is a field present? That's fast and reliable, but…