analiza aiqbench
This empirical audit demonstrates that standard per-channel linear projections in multi-channel transformers spontaneously achieve near-orthogonality, effectively recovering channel identity without explicit regularization or complex architectures. The study identifies specific failure modes for shared-scalar and overfitting-prone baselines while confirming that elaborate alternatives offer only marginal gains unless task-specific constraints demand them. Consequently, practitioners can confidently adopt the simpler default encoder by design rather than engineering complexity to solve a non-existent problem in most scenarios.
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
arXiv:2606.04752v3 Announce Type: replace-cross Abstract: Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step. We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an orthogonality regulariser, a nonlinear MLP, block-partitioned concatenation, channel-independent and channel-as-token architectures, and a projected positional encoding -- on a synthetic benchmark where channel identity is informative and on ETTh1, scored by next-step negative log-likelihood. The headline is practical near-equivalence within a wide "top tier": the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences. A direct geometric probe attributes this to a spontaneous orthogonalisation of the per-channel projections: with no explicit regulariser they tighten well beyond the near-orthogonality of random initialisation, letting the standard linear recover channel identity from the summed embedding. Two encoders lose decisively: the shared-scalar baseline collapses for information-theoretic reasons we make explicit, and the channel-independent PatchTST-spirit baseline overfits universally on the synthetic benchmark and underperforms on both. Paired tests resolve two small gaps: projecting the sinusoidal positional encoding through a learned linear layer edges the rest at small $C$ by extending this orthogonality to the positional subspace; a nonlinear MLP stem edges them at the largest $C$, with the gap shrinking under more training data. The practical recommendation: use the standard per-channel linear projection by default; reach for something more elaborate only when the task calls for it.
Abstract:Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step. We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an orthogonality regulariser, a nonlinear MLP, block-partitioned concatenation, channel-independent and channel-as-token architectures, and a projected positional encoding -- on a synthetic benchmark where channel identity is informative and on ETTh1, scored by next-step negative log-likelihood. The headline is practical near-equivalence within a wide "top tier": the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences. A direct geometric probe attributes this to a spontaneous orthogonalisation of the per-channel projections: with no explicit regulariser they tighten well beyond the near-orthogonality of random initialisation, letting the standard linear recover channel identity from the summed embedding. Two encoders lose decisively: the shared-scalar baseline collapses for information-theoretic reasons we make explicit, and the channel-independent PatchTST-spirit baseline overfits universally on the synthetic benchmark and underperforms on both. Paired tests resolve two small gaps: projecting the sinusoidal positional encoding through a learned linear layer edges the rest at small $C$ by extending this orthogonality to the positional subspace; a nonlinear MLP stem edges them at the largest $C$, with the gap shrinking under more training data. The practical recommendation: use the standard per-channel linear projection by default; reach for something more elaborate only when the task calls for it.
23 pages, 2 figure, 13 tables. Code: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.04752 [cs.LG]
(or arXiv:2606.04752v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 253 cuvinte
Subiecte conexe
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
arXiv · 5 iun. 2026
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
arXiv · 9 iun. 2026
News
Quantum Computing Report · 25 iul. 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026