aiqbench analysis
The article describes how a UK digital health company evaluates Nu, an LLM-based coaching agent in a regulated setting. Its main point is that single-turn fixtures and averaged scores miss conversational failures and can misattribute errors to the model when the rubric is at fault. For readers building agents, the useful takeaway is the shift to conversation-level evals and simulated multi-turn dialogue, though the article describes one team's approach rather than measured results.
How Numan Built an Eval-First AI Health Coach with LangGraph and LangSmith
How Numan built Nu, an AI health coach, on LangGraph and LangSmith with per-category evals, simulated conversations, and safety gates on a regulatory line.
Numan is a UK digital health company helping patients manage long-term conditions across weight, hormones, and sexual health. Many of our patients are on a weight-loss programme and use the app daily to track progress. When building Nu, our AI health coach, one of our design challenges was delivering high-quality health coaching without crossing the line into medical diagnosis, prescriptions, or making clinical decisions—all strict boundaries for agents in our highly regulated medical space. While we initially focused on the agent and automation around it, what turned out to matter most was the quality of our evaluation criteria. This is the story of how we developed and improved our evaluation criteria to deliver high-quality user experiences while steering clear of areas subject to regulation. We started where most teams start, with a dataset of inputs and expected outputs, and an LLM judge scoring each reply. While that playbook works well for a Q&A agent, coaching is a conversation, not an answer. Every approach that fixed Nu's reply in advance failed in two ways: 1. Single-turn evals on static fixtures (a fixed input, a fixed expected output, and a judge scoring one…
Excerpt shown under the publisher's RSS terms · 191 words · full text and figures are on the publisher's site