aiqbench analysis
This paper introduces AutoMMemo, a novel framework that enables multimodal agents to automatically design memory mechanisms for learning from heterogeneous experiences. By representing memory strategies as executable programs discovered through iterative search and reflection, the method significantly improves performance on GUI navigation and visual reasoning benchmarks without relying on manual schema design. The work addresses a critical gap in experience-driven learning by adapting memory architectures specifically for complex multimodal environments rather than textual settings alone.
Learning to Learn from Multimodal Experience
arXiv:2605.16857v2 Announce Type: replace Abstract: Multimodal agents increasingly interact with real-world-like environments and generate rich trajectories spanning perception, reasoning, and action. A key research focus is how to transform these raw multimodal trajectories into reusable experience that can improve future task performance. Existing experience-driven learning methods are mostly developed in textual settings and typically rely on manually designed memory schemas, making them less effective for multimodal experience that requires composing heterogeneous signals across modalities, temporal scales, and abstraction levels. We propose AutoMMemo, a general framework that enables agents to automatically design memory mechanisms for learning from multimodal experience. AutoMMemo represents each memory mechanism as an executable memo program, which defines how past episodes are stored, organized, and retrieved. It discovers effective memo programs through an iterative process of update-then-retrieve evaluation, reflection-guided mutation, and budget-aware tree search. Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show that AutoMMemo consistently improves agent performance over no-memory agents and fixed memory baselines, while maintaining reasonable token and interaction costs. Transfer and search-process analyses further demonstrate that the learned memory designs capture reusable experience structures and progressively refine retrieval relevance. These results highlight adaptive memory design as a key optimization target for multimodal experience-driven learning. Code is available at https://github.com/XingYuSSS/AutoMMemo.
Abstract:Multimodal agents increasingly interact with real-world-like environments and generate rich trajectories spanning perception, reasoning, and action. A key research focus is how to transform these raw multimodal trajectories into reusable experience that can improve future task performance. Existing experience-driven learning methods are mostly developed in textual settings and typically rely on manually designed memory schemas, making them less effective for multimodal experience that requires composing heterogeneous signals across modalities, temporal scales, and abstraction levels. We propose AutoMMemo, a general framework that enables agents to automatically design memory mechanisms for learning from multimodal experience. AutoMMemo represents each memory mechanism as an executable memo program, which defines how past episodes are stored, organized, and retrieved. It discovers effective memo programs through an iterative process of update-then-retrieve evaluation, reflection-guided mutation, and budget-aware tree search. Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show that AutoMMemo consistently improves agent performance over no-memory agents and fixed memory baselines, while maintaining reasonable token and interaction costs. Transfer and search-process analyses further demonstrate that the learned memory designs capture reusable experience structures and progressively refine retrieval relevance. These results highlight adaptive memory design as a key optimization target for multimodal experience-driven learning. Code is available at this https URL.
Artificial Intelligence (cs.AI)
arXiv:2605.16857 [cs.AI]
(or arXiv:2605.16857v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.16857 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 295 words
Related coverage
FinAcumen: Financial Multimodal Reasoning via Self-Evolving Experience Memory Harness
arXiv · Jun 17, 2026
ME-Brain-1.0: Memory, Cognition and Action for Evolving Embodied Intelligence
arXiv · Sep 22, 2026
FinAcumen: Financial Multimodal Reasoning via Self-Evolving Experience Memory Harness
arXiv · Aug 25, 2026
Paired Multimodal Scaling Laws
arXiv · Sep 30, 2026