aiqbench analysis
This research establishes that the "near-floor" geometry observed in trained overcomplete codes is not an artifact of fitting but a specific signature of learned leverage dispersion, distinguishing genuine model representations from random initializations. The study provides rigorous mathematical bounds showing that released decoder matrices consistently exceed theoretical efficiency floors, indicating that training dynamics actively shape feature interactions beyond simple linear readout constraints.
Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes
arXiv:2605.01192v2 Announce Type: replace Abstract: An overcomplete code packs $F$ features into $d<F$ dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor $W(F,d)=(F-d)/(d(F-1))$. Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is $\frac{d}{F(F-d)}(\sum_i h_i^{-1}-F)$ with $\sum_i h_i=d$, so it equals one precisely when the leverage $h_i$ is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: $L^2$ reaches $R_{geom}=18.5635$ to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for $L^4$, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.
Abstract:An overcomplete code packs $F$ features into $d<F$ dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor $W(F,d)=(F-d)/(d(F-1))$. Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is $\frac{d}{F(F-d)}(\sum_i h_i^{-1}-F)$ with $\sum_i h_i=d$, so it equals one precisely when the leverage $h_i$ is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: $L^2$ reaches $R_{geom}=18.5635$ to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for $L^4$, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.
v2: 53 pages, 7 figures. Substantial revision and extension; title changed. Three formal statements in v1 were too strong and are corrected here (two need h_i <= 1/2; one is wrong at kappa_i = 1), and two empirical readings are withdrawn. See "Changes from version 1" in the manuscript. Code: this http URL
Machine Learning (cs.LG); Information Theory (cs.IT)
arXiv:2605.01192 [cs.LG]
(or arXiv:2605.01192v2 [cs.LG] for this version)
Full text under the publisher's open licence · 346 words
