analiza aiqbench
This article presents CHERRY, an AI architecture that decouples token supervision from shared weights to achieve significant model compression while maintaining performance on specific tasks like Korean language processing. The research demonstrates a novel trade-off where selected-token supervision preserves discrimination capabilities even when free generation is constrained, offering insights into efficient training and sovereign LLM development for niche languages.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
arXiv:2606.31796v2 Announce Type: replace Abstract: Frontier language capability is usually bought with frontier compute; CHERRY shows a different trade. It is a sovereign Korean model family built on one principle: supervise the tokens that decide the answer, and let shared weights carry the rest. Under matched compute this exposes a sharp, reproducible dissociation---selected-token supervision preserves held-out discrimination yet collapses free generation, and a full-sequence anchor recovers only part of the gap. The same signal drives a heal-after-merge recurrent-representational-yield loop that collapses 48 layers to 6 unique blocks at near-dense parity (227M at loss 2.934 vs a 566M dense model at 2.926) and composes them by MoEE fusion (2.789)---a recurrent-compression direction independently pursued by concurrent frontier looped-MoE work, which we project (not yet measure) to frontier scale. It also installs metacognition from two-token supervision (200 held-out KO prompts/type, kappa>0.82, +/-6.9pp): self-correction 12->47% and jailbreak 23->4% at 97.6% loss-retention on 1.2B, with a pre-registered 1B->13.7B ablation localising the operand-binding limit to capacity (1B lookup vs 13.7B H-PRESERVE); and it specializes a 1.8B model to reported human-expert level on CyberMetric (75.0% vs a 30-expert average 72.24%). The released 1.8B tokenizer is 9.2% more Korean-efficient than Gemma-4; the from-scratch 12B adds a sovereign Korean tokenizer (vocab 131,037). On the government-operated K-AI Korean-LLM leaderboard the released 1.8B leads the HLE(Ko) column (0.123 vs 0.077; overall 51/78). At the frontier we fine-tune and serve an adapted 122B model on one 120GB accelerator (83GB measured peak). Provenance differs by member and we state it exactly; the architecture and recipe are ours throughout, and every claim is bound to a released curated measurement.
Abstract:Frontier language capability is usually bought with frontier compute; CHERRY shows a different trade. It is a sovereign Korean model family built on one principle: supervise the tokens that decide the answer, and let shared weights carry the rest. Under matched compute this exposes a sharp, reproducible dissociation---selected-token supervision preserves held-out discrimination yet collapses free generation, and a full-sequence anchor recovers only part of the gap. The same signal drives a heal-after-merge recurrent-representational-yield loop that collapses 48 layers to 6 unique blocks at near-dense parity (227M at loss 2.934 vs a 566M dense model at 2.926) and composes them by MoEE fusion (2.789)---a recurrent-compression direction independently pursued by concurrent frontier looped-MoE work, which we project (not yet measure) to frontier scale. It also installs metacognition from two-token supervision (200 held-out KO prompts/type, kappa>0.82, +/-6.9pp): self-correction 12->47% and jailbreak 23->4% at 97.6% loss-retention on 1.2B, with a pre-registered 1B->13.7B ablation localising the operand-binding limit to capacity (1B lookup vs 13.7B H-PRESERVE); and it specializes a 1.8B model to reported human-expert level on CyberMetric (75.0% vs a 30-expert average 72.24%). The released 1.8B tokenizer is 9.2% more Korean-efficient than Gemma-4; the from-scratch 12B adds a sovereign Korean tokenizer (vocab 131,037). On the government-operated K-AI Korean-LLM leaderboard the released 1.8B leads the HLE(Ko) column (0.123 vs 0.077; overall 51/78). At the frontier we fine-tune and serve an adapted 122B model on one 120GB accelerator (83GB measured peak). Provenance differs by member and we state it exactly; the architecture and recipe are ours throughout, and every claim is bound to a released curated measurement.
64pp, LaTeX. v2 rebuilds arXiv:2606.31796 into a full report: matched-compute discrimination/generation dissociation, recurrent-yield compression (48->6)+MoEE, pre-registered 1B->13.7B H-PRESERVE, sovereign Korean tokenizer (+9.2% vs Gemma-4), government HLE(Ko) column lead, cyber specialization. Recurrent compression cited by Loopie (arXiv:2607.16051). Tables incl.; v1 in history
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.31796 [cs.CL]
(or arXiv:2606.31796v2 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 316 cuvinte
Subiecte conexe
ScoutNeRV: Rapid Encoding of Grid-Based Video INRs via ScoutNet
arXiv · 24 sept. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
SMI: Ontology-Grounded Semantic IDs for Hierarchy-Aware Medical Concept Representation
arXiv · 15 oct. 2026
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · 8 oct. 2026