What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Title:What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Abstract:The information bottleneck (IB) trades compression of an input $X$ against prediction of a target $Y$ through a multiplier $\beta=1/\lambda$, yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as $A(\beta)=D(\beta)W$: the information-bearing directions $W$ do not depend on $\beta$, and $\beta$ acts only through scalar gains. We ask whether this separation of \emph{representation} from \emph{rate control} survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation \(F(X)\), the \emph{separation gap} is controlled by the \emph{sufficiency deficit} $\epsilon_F=I(X;Y\mid F(X))$: We show there is at least $\epsilon_F$ in the worst case for finite inputs, and it is bounded uniformly in $\lambda$ by a function of $\epsilon_F$ for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by $7.7\times$ and wall-clock time by about $2\times$ relative to end-to-end VIBs, and attains higher budget-averaged accuracy ($65.6\%$ versus $57.3\%$ per domain; $67.1\%$ versus $57.1\%$ pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression ($\lambda\le0.036$), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.

Machine Learning (cs.LG); Information Theory (cs.IT)

arXiv:2610.36255 [cs.LG]

(or arXiv:2610.36255v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.36255 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.