aiqbench analysis
This research proposes a scalable paradigm for foundation models that bypasses text extraction to directly leverage visual information from documents and web pages. By training on raw images rather than converted plain text, the approach captures rich contextual details like equations and layouts that are often lost in traditional pipelines. The study demonstrates that this unsupervised visual pretraining consistently outperforms standard text-only methods across various backbones, offering a more efficient path to advanced language intelligence.
Scalable Visual Pretraining for Language Intelligence
arXiv:2607.09657v2 Announce Type: replace-cross Abstract: The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.
Abstract:The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM)
arXiv:2607.09657 [cs.CV]
(or arXiv:2607.09657v2 [cs.CV] for this version)
Full text under the publisher's open licence · 165 words
Related coverage
Building a Foundation Stack for General-Purpose Robots
IEEE Spectrum · Jul 13, 2026
Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding
arXiv · Oct 8, 2026
OmniEcho: Spatial Audio Understanding for Embodied Agents
arXiv · Sep 22, 2026
NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model
NVIDIA · Apr 28, 2026