Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

At our latest Snorkel AI Reading Group, Yijia Shao (Stanford NLP) stopped by our San Francisco office to present Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. As LLM agents get better at automating tasks on their own, a large class of real-world problems still needs a human in the loop – for their preferences, their domain expertise, or simply for control. Co-Gym is an open framework for building and evaluating agents that communicate bidirectionally with humans while working inside a task environment, using a flexible, non-turn-taking interaction paradigm, with an evaluation suite that measures both the outcome of a collaboration and the process behind it. Across travel planning, related-work writing, and tabular analysis, the best collaborative agents consistently beat their fully autonomous counterparts when evaluated by real users – and Yijia is candid about where today’s models still fall short, especially on communication and situational awareness. Lightly edited for readability. Thank you so much for the introduction, and hi everyone, thank you for having me. It’s my first time visiting Snorkel’s office; it’s a really nice place, and it’s great to see a full room, with more people than I expected. I’ll start with a quick introduction.…