aiqbench analysis
This research investigates knowledge transfer across dialogue games using Large Language Models, revealing that visuospatial tasks benefit most from cross-game training while similarity-based metrics fail to capture actual transferability patterns. The findings suggest that current vector-space approximations are insufficient for modeling the complex cognitive skills required in interactive multi-agent environments. For researchers in machine learning, this highlights the need for more sophisticated evaluation metrics when assessing generalization in specialized dialogue benchmarks.
Investigating Knowledge Transfer Across Interactive Dialogue Games
arXiv:2608.23969v1 Announce Type: new Abstract: Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.
Abstract:Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.
Computation and Language (cs.CL)
arXiv:2608.23969 [cs.CL]
(or arXiv:2608.23969v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.23969 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 10, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
LessWrong · Oct 2, 2026
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble
LessWrong · Sep 21, 2026