KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

Abstract:Residual connections are foundational to modern neural networks, yet the standard additive update provides no explicit mechanism for structured inter-channel interaction. We introduce KITINet (KInetics Theory Inspired Network), a training-time module with no additional trainable parameters that augments residual updates with stochastic pairwise feature interactions inspired by collision sampling in kinetic theory. KITINet reshapes channel groups as particles and mixes their features according to relative distance and velocity, while leaving the underlying residual layer trainable in the usual way. The module is disabled at inference, so the deployed model is identical to the original backbone and incurs no additional inference cost. Across language pre-training and fine-tuning, image classification, and PDE operator learning, KITINet demonstrates broad and robust performance gains across diverse tasks and architectures. We further observe enhanced parameter condensation across several settings and develop a simplified stochastic analysis that provides a mechanistic account of how collision-inspired interactions can promote condensation dynamics.

18 pages, 10 figures, 9 tables. Substantially revised and expanded version. We revise the title and presentation, clarify KITINet as a parameter-free training-time residual interaction module with zero additional inference cost, add a physics-motivated and theoretical analysis of parameter condensation, add language-model pre-training, continued pre-training, and LoRA fine-tuning experiments

Machine Learning (cs.LG)

arXiv:2505.17919 [cs.LG]

(or arXiv:2505.17919v2 [cs.LG] for this version)