aiqbench analysis
This research investigates the feasibility of composing frozen language models from different architectural families using a shared latent adapter to reduce computational costs. While the method successfully lowers the Transformer key-value cache by over 84% with minimal accuracy loss on specific benchmarks, it fails to match the performance of the superior parent model on general language modeling tasks outside its training domain. The study concludes that while intermediate representations can be mapped between architectures under strict conditions, a universal shared semantic space across diverse model families does not currently exist.
NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space
arXiv:2609.38261v1 Announce Type: cross Abstract: In this paper we propose NinaXander, a series of composed language models obtained by connecting layers of frozen language models from different architecture families with a single trained shared-latent adapter. A composed model runs the first layers of one model, converts the resulting intermediate representation once with the adapter, and then runs the remaining layers of the other model. Once the adapter is trained, several composed models that connect at different layers are obtained without retraining. Using the recurrent RWKV-4-Raven-7B and the Transformer-based Tulu-Pythia-6.9b, abbreviated as RWKV and Pythia, this study examines whether frozen models from different families can be recombined post hoc. The composed models answered multiple-choice questions, and those whose generations we examined produced syntactically well-formed text. The configuration that combines the first 5 layers of Pythia with the remaining 27 layers of RWKV reduced the Transformer key-value (KV) cache by 84.4% with accuracy not significantly different from that of RWKV alone. In multiple-choice accuracy, however, no composed model matched the parent model Pythia, and language-modeling performance decreased sharply on WikiText, a corpus of Wikipedia articles outside the training domain. The correspondence between intermediate representations was also obtained in one favorable case, with a shared tokenizer, the same depth, and the same hidden width, and does not show that the models share a general semantic space.
Abstract:In this paper we propose NinaXander, a series of composed language models obtained by connecting layers of frozen language models from different architecture families with a single trained shared-latent adapter. A composed model runs the first layers of one model, converts the resulting intermediate representation once with the adapter, and then runs the remaining layers of the other model. Once the adapter is trained, several composed models that connect at different layers are obtained without retraining. Using the recurrent RWKV-4-Raven-7B and the Transformer-based Tulu-Pythia-6.9b, abbreviated as RWKV and Pythia, this study examines whether frozen models from different families can be recombined post hoc. The composed models answered multiple-choice questions, and those whose generations we examined produced syntactically well-formed text. The configuration that combines the first 5 layers of Pythia with the remaining 27 layers of RWKV reduced the Transformer key-value (KV) cache by 84.4% with accuracy not significantly different from that of RWKV alone. In multiple-choice accuracy, however, no composed model matched the parent model Pythia, and language-modeling performance decreased sharply on WikiText, a corpus of Wikipedia articles outside the training domain. The correspondence between intermediate representations was also obtained in one favorable case, with a shared tokenizer, the same depth, and the same hidden width, and does not show that the models share a general semantic space.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.38261 [cs.CL]
(or arXiv:2609.38261v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.38261 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 314 words
Related coverage
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
LessWrong · Oct 2, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
Small LLMs perform no-CoT two-hop reasoning on synthetic facts with real bridges
LessWrong · Oct 17, 2026