analiza aiqbench
This research proposes a novel architecture that decouples visual perception from the language model by using a deterministic semantic serializer to translate raw sensor data into text-aligned decisions. The study demonstrates that this approach allows smaller, text-only LLMs (down to 1.5B parameters) to outperform standard zero-shot vision-language models on embodied robotics tasks without modifying the core reasoning engine. Ultimately, the work suggests a viable path for deploying efficient AI agents in resource-constrained environments by separating high-fidelity perception from lightweight reasoning capabilities.
Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models
arXiv:2609.29601v1 Announce Type: new Abstract: End-to-end vision-language models (VLMs) bind visual competence to the scale of their language model: as the language model shrinks, perception and reasoning degrade together. We study an embodied scene question-answering (QA) interface in which vision never enters the language model. A frozen perception stack detects and ranges objects; a deterministic semantic serializer compiles the perceived state, errors included, into decision-aligned text; an unmodified text-only large language model (LLM) answers. On a visible-scope-matched, occlusion-audited campus-robot benchmark, under a prospectively frozen criterion, the serialized interface, using detectors fine-tuned in-domain within each fold, outperforms a zero-shot VLM whose language model has the same 7B scale (0.7892 vs 0.7462), with a larger margin at 3B (0.7673 vs 0.6913). Preregistered decoupling experiments show the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope. The advantage grows as the reader shrinks to 1.5B and reverses at 0.5B, and a ground-truth oracle locates the reader-capability floor. Under matched task supervision the interfaces converge: a VLM fine-tuned with low-rank adaptation (LoRA) overtakes the zero-shot system but only ties an equally supervised text reader (0.8441 vs 0.8396, no statistically resolved difference), and both routes remain perception-bound. Reported perception parameters are comparable to those of the VLM's vision tower, and total compute is not smaller.
Abstract:End-to-end vision-language models (VLMs) bind visual competence to the scale of their language model: as the language model shrinks, perception and reasoning degrade together. We study an embodied scene question-answering (QA) interface in which vision never enters the language model. A frozen perception stack detects and ranges objects; a deterministic semantic serializer compiles the perceived state, errors included, into decision-aligned text; an unmodified text-only large language model (LLM) answers. On a visible-scope-matched, occlusion-audited campus-robot benchmark, under a prospectively frozen criterion, the serialized interface, using detectors fine-tuned in-domain within each fold, outperforms a zero-shot VLM whose language model has the same 7B scale (0.7892 vs 0.7462), with a larger margin at 3B (0.7673 vs 0.6913). Preregistered decoupling experiments show the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope. The advantage grows as the reader shrinks to 1.5B and reverses at 0.5B, and a ground-truth oracle locates the reader-capability floor. Under matched task supervision the interfaces converge: a VLM fine-tuned with low-rank adaptation (LoRA) overtakes the zero-shot system but only ties an equally supervised text reader (0.8441 vs 0.8396, no statistically resolved difference), and both routes remain perception-bound. Reported perception parameters are comparable to those of the VLM's vision tower, and total compute is not smaller.
12 pages, 2 figures. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
Robotics (cs.RO); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2609.29601 [cs.RO]
(or arXiv:2609.29601v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.29601 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 272 cuvinte
Subiecte conexe
What can language models teach us about understanding?
LessWrong · 9 oct. 2026
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
arXiv · 28 aug. 2026
QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models
arXiv · 3 aug. 2026
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026