analiza aiqbench
This paper introduces Kan Extension Transformers (KETs), a novel categorical framework that unifies diverse Transformer architectures under a single mathematical design language. By formalizing layers as weighted structured extension operators, the research clarifies conditions for strict autoregression and evaluates performance across multiple causal regimes on standard text datasets. The findings suggest that gains in self-conditioned models stem primarily from information flow mechanisms rather than neighborhood topology alone.
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
arXiv:2605.27259v2 Announce Type: replace Abstract: We propose Kan Extension Transformers (KETs) as a categorical design language for a diverse group of Transformer implementations. A layer can be viewed generally as a weighted structured extension operator: attention uses token neighborhoods, geometric mixing uses sparse incidences, and KET uses simplicial sources. This operator is an actual enriched left Kan extension only when the source values are functorial, the weights are representable hom-objects (or the specified profunctor action), and aggregation realizes the corresponding coend; otherwise ``Kan-style'' denotes an interpretation rather than an identity theorem. Predict-detach blocks gradients through a predictive carrier and avoids transporting teacher-forced hidden states, but detach alone does not make a noncausal update strictly autoregressive: every carrier consumed at target $t$ must also be measurable from the prefix available at $t$. We evaluate 12 implementations on Penn Treebank, WikiText-2, and WikiText-103 across strict-causal and self-conditioned regimes, using widths $d=64,256$ and depths $L=2,8,16$ across the reported studies. Quadratic KET is strongest among the compared strict-causal architectures on WikiText-2 and WikiText-103; the largest cross-regime gains arise from additional self-conditioning information, not neighborhood design alone.
Abstract:We propose Kan Extension Transformers (KETs) as a categorical design language for a diverse group of Transformer implementations. A layer can be viewed generally as a weighted structured extension operator: attention uses token neighborhoods, geometric mixing uses sparse incidences, and KET uses simplicial sources. This operator is an actual enriched left Kan extension only when the source values are functorial, the weights are representable hom-objects (or the specified profunctor action), and aggregation realizes the corresponding coend; otherwise ``Kan-style'' denotes an interpretation rather than an identity theorem. Predict-detach blocks gradients through a predictive carrier and avoids transporting teacher-forced hidden states, but detach alone does not make a noncausal update strictly autoregressive: every carrier consumed at target $t$ must also be measurable from the prefix available at $t$. We evaluate 12 implementations on Penn Treebank, WikiText-2, and WikiText-103 across strict-causal and self-conditioned regimes, using widths $d=64,256$ and depths $L=2,8,16$ across the reported studies. Quadratic KET is strongest among the compared strict-causal architectures on WikiText-2 and WikiText-103; the largest cross-regime gains arise from additional self-conditioning information, not neighborhood design alone.
Clarified that predict-detach is strictly autoregressive only under a target-relative prefix-measurability condition; restricted exact Kan-extension claims to representable enriched cases; and clarified experimental coverage across depths 2, 8, 16 and widths 64, 256. A detailed change log appears in the supplement
Machine Learning (cs.LG)
arXiv:2605.27259 [cs.LG]
(or arXiv:2605.27259v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 229 cuvinte
Subiecte conexe
Build an explainable next-best-product recommendation system for banking on AWS
AWS · 24 iul. 2026
Tencent Open-Sources AngelSpec: A Unified Training Framework for MTP and Block-Parallel Speculative Decoding on Hy3 Models
MarkTechPost · 30 iul. 2026
DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoning
arXiv · 17 iun. 2026
Exploring Diffusion Transformers for Cross-Modal Augmentation in Multimodal Brain State Decoding
arXiv · 12 sept. 2026