aiqbench analysis
SkillLens introduces a Visual Skill Card schema that enhances GUI agent performance by enabling on-policy distillation from illustrated procedures, eliminating the need for retrieval at deployment. The approach demonstrates significant improvements in step success rates and action accuracy compared to text-only methods across diverse web tasks. This work advances the field of embodied AI by formalizing how visual context can be distilled into compact student models for autonomous interface navigation.
SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
arXiv:2608.10775v2 Announce Type: replace Abstract: Graphical user interface (GUI) agents can reuse past interaction experience. We study how illustrated procedures support action prediction and students deployed without retrieval. We introduce SkillLens, which represents procedures, applicability cues, and interface views in a shared Visual Skill Card (VSC) schema. Retrieved cards provide a frozen executor with reference evidence alongside the current screen. CardDistill uses this schema to construct privileged teacher context from labeled training steps for on-policy distillation; the student learns from native inputs and requires no retrieval at deployment. With matched training data and update counts, students trained with full-card teachers improve step success rate by an average of 5.04 percentage points over those trained with compact-text teachers across three training seeds on 119 tasks from ten websites absent from training. An independent runtime control uses training-source memory and pre-action retrieval queries: full cards achieve 12/64 correct actions (18.75%) versus 4/64 (6.25%) for text-only cards. These results establish the value of the complete VSC teacher-context package for retrieval-free students and provide evidence for pre-action retrieval of illustrated procedures.
Abstract:Graphical user interface (GUI) agents can reuse past interaction experience. We study how illustrated procedures support action prediction and students deployed without retrieval. We introduce SkillLens, which represents procedures, applicability cues, and interface views in a shared Visual Skill Card (VSC) schema. Retrieved cards provide a frozen executor with reference evidence alongside the current screen. CardDistill uses this schema to construct privileged teacher context from labeled training steps for on-policy distillation; the student learns from native inputs and requires no retrieval at deployment. With matched training data and update counts, students trained with full-card teachers improve step success rate by an average of 5.04 percentage points over those trained with compact-text teachers across three training seeds on 119 tasks from ten websites absent from training. An independent runtime control uses training-source memory and pre-action retrieval queries: full cards achieve 12/64 correct actions (18.75%) versus 4/64 (6.25%) for text-only cards. These results establish the value of the complete VSC teacher-context package for retrieval-free students and provide evidence for pre-action retrieval of illustrated procedures.
Artificial Intelligence (cs.AI)
arXiv:2608.10775 [cs.AI]
(or arXiv:2608.10775v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.10775 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words