analiza aiqbench
HYDRA-X advances unified multimodal models by integrating image and video tokenization into a single Vision Transformer, addressing key challenges in spatiotemporal reconstruction and semantic awareness. The model introduces efficient frame-level causal attention and hierarchical compression techniques that outperform existing methods while improving editing consistency at the latent level. This research provides a foundational architecture for next-generation systems capable of natively processing diverse visual inputs without separate encoders.
HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers
arXiv:2606.13289v1 Announce Type: cross Abstract: Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.
Abstract:Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2606.13289 [cs.CV]
(or arXiv:2606.13289v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2606.13289 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 224 cuvinte
Subiecte conexe
X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining
arXiv · 16 iun. 2026
MoTIF-X: A Multimodal Tokenized Framework for Interpretable and Extensible Molecular Representation Learning
arXiv · 30 sept. 2026
Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference
arXiv · 31 oct. 2026
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
arXiv · 9 sept. 2026