Zebra-HyLo: Upcycling Transformers into Long-Context Hybrid LLMs on AMD Instinct™ GPUs

September 22, 2026 by Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh, Aref Jafari, Akash Haridas, Mingyu Yang, Vansh Bhatia, Guihong Li, Vikram Appia, Emad Barsoum.

16 min read. | 3813 total words.

Hybrid language models that interleave attention with linear sequence-modeling blocks have become the default answer to the cost of long context. Jamba, Samba, Qwen3-Next and Kimi-Linear all take this shape, and they all share one property that makes them expensive to adopt: they are pretrained from scratch. Every one of them pays the full cost of building a foundation model again, which means the enormous investment already sunk into existing Transformer checkpoints is thrown away.

Upcycling is the alternative. Instead of pretraining a hybrid, you take a pretrained Transformer, replace its attention layers with cheaper block types, and spend a small amount of post-training teaching the new blocks to behave like the old ones. Prior work in this direction — MambaInLlama [1], Llamba [4], MOHAWK, and our own Zebra-Llama [3] — showed this works, but measured success almost entirely on short-context benchmarks like perplexity and commonsense reasoning accuracy. That leaves the obvious question unanswered. Hybrid architectures are motivated by long sequences, so does an upcycled hybrid actually inherit long-context ability from its Transformer parent?

The short answer we found is no, not unless you train for it explicitly. A model upcycled at 8K context scores 55.1 on RULER at 8K and 0.8 at 64K — it has essentially no usable long-context behaviour, despite an architecture built for exactly that.

Zebra-HyLo (hybrid long-context) is our recipe for fixing this. It treats long-context preservation as a first-class training objective rather than something the architecture provides for free, and it delivers models that extend usable context by up to 32x, cut KV-cache memory by more than 90%, and serve up to 2M tokens in vLLM on eight AMD Instinct™ MI300X GPUs — all through post-training, on roughly 10B tokens, with no pretraining from scratch. Figure 1 below previews the headline result across context lengths.

In this blog you will learn how to turn a pretrained Transformer into a long-context hybrid model on AMD Instinct™ MI300X GPUs. You will see how to initialize MLA and linear blocks from existing weights, how to stage distillation from 2K out to 64K context, how to make 64K teacher-guided distillation fit in GPU memory, and how to serve the resulting model in vLLM. Along the way you get the 14 released checkpoints, the exact training configs behind every one of them, and the evaluation numbers to compare against. Ready to upcycle your own checkpoint? Clone the repository and follow along — the whole recipe runs on a single eight-GPU node.

Figure 1: Short-context math performance and average RULER accuracy across 8K, 16K, 32K and 64K context lengths. Zebra-HyLo models stay competitive on short context while clearly outperforming upcycling baselines on long context, within a limited upcycling data budget. Source: Figure 1 from the HyLo paper.#

Figure 1: Short-context math performance and average RULER accuracy across 8K, 16K, 32K and 64K context lengths. Zebra-HyLo models stay competitive on short context while clearly outperforming upcycling baselines on long context, within a limited upcycling data budget. Source: Figure 1 from the HyLo paper.#

Long-context-aware upcycling. We extend the Zebra-Llama recipe so that long-context ability survives conversion, instead of optimizing short-context accuracy alone.

Long-context-aware upcycling. We extend the Zebra-Llama recipe so that long-context ability survives conversion, instead of optimizing short-context accuracy alone.

Staged context expansion. Prior upcycling work trains to roughly 24K context. We stage training from 2K through 8K to 64K and show how the training sequence length, not the architecture, determines long-context generalization.

Staged context expansion. Prior upcycling work trains to roughly 24K context. We stage training from 2K through 8K to 64K and show how the training sequence length, not the architecture, determines long-context generalization.

Teacher-guided long-context distillation. Chunk-wise KL supervision from an 8B teacher at 64K context, which required a stack of memory optimizations to be feasible at all.

Teacher-guided long-context distillation. Chunk-wise KL supervision from an 8B teacher at 64K context, which required a stack of memory optimizations to be feasible at all.

Two backbones, two linear blocks. Llama-3.2-1B/3B and Qwen3-1.7B, each with Gated DeltaNet (GDN) or Mamba-2 (M2), showing the recipe is not tied to one architecture family.

Two backbones, two linear blocks. Llama-3.2-1B/3B and Qwen3-1.7B, each with Gated DeltaNet (GDN) or Mamba-2 (M2), showing the recipe is not tied to one architecture family.

Deployment-oriented. A vLLM integration that interleaves Mamba/GDN state with MLA KV cache, serving contexts up to 2M tokens where the Llama-3.2-3B baseline runs out of memory past 64K.

Deployment-oriented. A vLLM integration that interleaves Mamba/GDN state with MLA KV cache, serving contexts up to 2M tokens where the Llama-3.2-3B baseline runs out of memory past 64K.

Everything is open. Training code, the exact configs behind all 14 released checkpoints, and the models themselves.

Everything is open. Training code, the exact configs behind all 14 released checkpoints, and the models themselves.

We release 14 checkpoints, all of them in the HyLo collection on Hugging Face. Names encode the layer layout: 4MLA12GDN is 4 MLA layers plus 12 Gated DeltaNet layers. "KV cache reduction" is how much of the backbone's KV-cache footprint the upcycled model gives back at the same sequence length — every checkpoint frees at least 92% of it.

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-1B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Llama-3.2-3B-Instruct

Table 1: The HyLo model family.

Every checkpoint is Apache-2.0 licensed and ships as a single model.safetensors alongside a hybrid_config.json that records its layer plan, so the loader can rebuild the right mix of MLA and linear layers without being told. Each model card carries the full per-task numbers and the exact training recipe for that checkpoint. One licensing caveat worth reading before you build on the 64K models: the long-context portion of the training mixture derives from nvidia/ChatQA2-Long-SFT-data, which is non-commercial, and that constrains models you train with the full mixture regardless of the Apache-2.0 licence on the code.

The KV cache is what makes long context expensive. A standard Transformer stores keys and values for every token in every layer, so memory grows linearly with sequence length and multiplies by depth. Llama-3.2-3B has 28 attention layers, and their combined cache is what makes it run out of memory past 64K in our serving benchmarks.

Multi-head Latent Attention (MLA), following DeepSeek-V3 and the MLA design introduced in DeepSeek-V2 [7], keeps real attention but caches a low-rank latent instead of full keys and values. Per token, the cache shrinks from 2 · H_kv · d_h to r_kv + d_rope. It preserves precise token-to-token retrieval at a fraction of the memory.

Multi-head Latent Attention (MLA), following DeepSeek-V3 and the MLA design introduced in DeepSeek-V2 [7], keeps real attention but caches a low-rank latent instead of full keys and values. Per token, the cache shrinks from 2 · H_kv · d_h to r_kv + d_rope. It preserves precise token-to-token retrieval at a fraction of the memory.

Linear blocks — Mamba-2 [5] or Gated DeltaNet [6] — carry a fixed-size recurrent state and cache nothing. Their memory is const