aiqbench analysis
The DeepSpeak-Agentic dataset offers a critical benchmark for evaluating forensic identification of AI agents and analyzing human-agent interaction dynamics. By providing over 37 hours of semi-structured audiovisual conversations, it enables rigorous testing of detection systems for AI-generated voices and faces in embodied agents. Additionally, the accompanying scalable data-capture system streamlines the creation and pairing of agents with human workers for standardized scenario testing.
The DeepSpeak-Agentic Dataset
arXiv:2606.03686v1 Announce Type: new Abstract: We present DeepSpeak-Agentic, a dataset of videos comprising over 37 hours of semi-structured conversations between a human and an embodied AI agent. We use this dataset to evaluate the automatic forensic identification (audio, video, or text) of AI agents, study the nature of human-agent interactions, and provide a benchmark for future advances in the large-language models and AI-generated voices and faces that power embodied AI agents. We also contribute a scalable data-capture system that creates agents, automatically pairs them with human crowd workers, records audiovisual conversations across specified scenarios, and identifies and separates the human and agent in the combined stream.
Abstract:We present DeepSpeak-Agentic, a dataset of videos comprising over 37 hours of semi-structured conversations between a human and an embodied AI agent. We use this dataset to evaluate the automatic forensic identification (audio, video, or text) of AI agents, study the nature of human-agent interactions, and provide a benchmark for future advances in the large-language models and AI-generated voices and faces that power embodied AI agents. We also contribute a scalable data-capture system that creates agents, automatically pairs them with human crowd workers, records audiovisual conversations across specified scenarios, and identifies and separates the human and agent in the combined stream.
Artificial Intelligence (cs.AI)
arXiv:2606.03686 [cs.AI]
(or arXiv:2606.03686v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.03686 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 195 words