Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation

Zhou Yu: Today I'm going to talk about simulation-driven AI agent testing and improvement. A little bit of background. I'm the co-founder of Arklex AI. I'm also a professor at Columbia University. I've been working on AI for quite a long time. I've been a professor for nine years. I've been working with various open-source models for pre-training, post-training, and so on. One of the things I really cared about is how do these models actually get into production? How can we actually utilize these models' abilities to actually perform real things that changes people's lives? One of the things that we noticed today and we have seen in a lot of reviews is like 95% of agents are staying in demos. They're very interesting to play with, they're very nice demos to look at, but they can't really create real values, and really making changes within the process we have been experiencing for many years. I'm going to talk about a couple of applications we have been working on before, either in my lab and/or in my startup. My startup is a spinoff of my lab's research, specifically on simulations. There are so many different kinds of agents. Today…