Presentation: Building Evals for AI Adoption: from Principles to Practice

Mallika Rao: How many of you have test datasets that are most definitely 100% wholesome and 100% covered for the products that you're building? I'm Mallika Rao. I have led search infrastructure teams at Twitter, trillions of documents, sub-50 millisecond latency budgets at global scale, highly personalized ranked search formats. We took it from just an inverted keyword lexical search to a highly personalized search at Twitter. Then I was at Walmart, where we built for a premium subscription service, a rewards model to boost growth, commerce, retention, acquisition, and all that good stuff. Most recently, the content systems at Netflix, where we process billions of personalization decisions every day for global scale, including recommendations, ranking, slicing the data, at scale. I've shipped intelligent systems that can scale up to over a billion users. Here's what I have learned at scale. Very rarely do the models actually come in the way of shipping products that thrive. It's actually your evaluation frameworks that can break your products, break your pipelines, and actually touch that user trust, which is so critical for shipping AI products at scale. Today, I want to talk about something that's invisible to your dashboards, but deadly to…