Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows

arXiv:2507.18405v3 Announce Type: replace-cross Abstract: Vision Transformers (ViTs) face two limitations: the rigid resolution dependency of positional embeddings, which complicates cross-resolution fine-tuning, and the quadratic complexity of attention. While Swin Transformer alleviates the latter through window attention, it suffers from fine-tuning. Following the philosophy "no token is an island," we present Iwin Transformer, a position-embedding-free hierarchical vision transformer that couples interleaved window attention with depthwise convolution inside a single block. Attention captures long-range dependencies, while convolution links local neighbors and implicitly encodes spatial position. This design not only reduces the quadratic complexity of attention but also enables two types of scalability: fine-tuning from low to high resolution and weight transfer from 2D to 3D. With window-size adjustment alone, direct $224^2{\rightarrow}384^2$ fine-tuning lifts Iwin-L from 86.4\% to 87.4\% top-1 accuracy on ImageNet-1K. Transferring an ImageNet-pretrained Iwin-T to video achieves 79.1\% on Kinetics-400, outperforming Swin-T (78.8\%) with 15.9\% fewer FLOPs. Iwin also remains competitive on ADE20K segmentation and class-conditional image generation (FlashDiT). Overall, Iwin offers an effective approach to simultaneously tackling the complexity and scalability challenges in ViTs. Code and models are at https://github.com/cominder/Iwin-Transformer.

Abstract:Vision Transformers (ViTs) face two limitations: the rigid resolution dependency of positional embeddings, which complicates cross-resolution fine-tuning, and the quadratic complexity of attention. While Swin Transformer alleviates the latter through window attention, it suffers from fine-tuning. Following the philosophy "no token is an island," we present Iwin Transformer, a position-embedding-free hierarchical vision transformer that couples interleaved window attention with depthwise convolution inside a single block. Attention captures long-range dependencies, while convolution links local neighbors and implicitly encodes spatial position. This design not only reduces the quadratic complexity of attention but also enables two types of scalability: fine-tuning from low to high resolution and weight transfer from 2D to 3D. With window-size adjustment alone, direct $224^2{\rightarrow}384^2$ fine-tuning lifts Iwin-L from 86.4\% to 87.4\% top-1 accuracy on ImageNet-1K. Transferring an ImageNet-pretrained Iwin-T to video achieves 79.1\% on Kinetics-400, outperforming Swin-T (78.8\%) with 15.9\% fewer FLOPs. Iwin also remains competitive on ADE20K segmentation and class-conditional image generation (FlashDiT). Overall, Iwin offers an effective approach to simultaneously tackling the complexity and scalability challenges in ViTs. Code and models are at this https URL.

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

arXiv:2507.18405 [cs.CV]

(or arXiv:2507.18405v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2507.18405 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows

Related coverage