Presentation: Building Reusable Evaluation Frameworks for Agentic AI Products

Susan Chang: My name is Susan. I'm a principal data scientist at Elastic. We make tools like Elasticsearch and Kibana. I've also written a book with O'Reilly called "Machine Learning Interviews." Today we're going to be talking about a few things. I'm going to talk a bit about what kind of agents we've been building and try to maybe connect them to what you folks might have been building. Tracing, which is, in my opinion, very important and a huge foundation, as well as how we use that to enable all these evaluations. I'm going to talk about how our company made this journey through maybe individual, like rebuilding a lot of evaluations towards building a shared framework, as well as what are the building blocks we have in that shared framework. Then tying it all together, and then see what takeaways that we have there. Just so that we can be roughly on the same page, I'm going to describe a bit of why we're building agents at Elastic and the type of agents that we're running in production. Elastic, we make Elasticsearch, so a lot of companies, they use us for just search or retrieval. They could use…