VISTA: A Controllable Platform for Generating and Auditing Egocentric Assistance Scenarios

Abstract:Evaluating whether AI agents can proactively assist humans in daily activities, ranging from routine household tasks to urgent safety-critical situations, requires diverse visual data. However, collecting such scenarios in the real world is often difficult, costly, or unsafe, and simulation environments often lack the social commonsense needed to simulate the consequences of different actions. In this work, we present VISTA, a controllable platform that uses a user-provided scenario seed, defined as a short natural-language description of the intended assistance situation, to generate editable plans, egocentric videos, and an auditable review trail. VISTA structures scenario intent around three interaction modes, including reactive, explicit proactive, and implicit proactive, and two consequence families, including safety-critical and everyday inconvenience, with no-assistance cases as controls. Its six-stage pipeline exposes the design brief, timed event script, first-frame plan, motion plan, and video plan, allowing users to revise each artifact in natural language before explicitly authorizing media generation. A human evaluation shows that videos retained by the complete VISTA workflow align more closely with their scenario seeds than outputs from two one-pass baselines. VISTA thereby makes targeted egocentric scenario generation inspectable, revisable, and empirically auditable.

Computation and Language (cs.CL)

arXiv:2605.10579 [cs.CL]

(or arXiv:2605.10579v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.10579 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.