aiqbench analysis
This research proposes a theoretical framework linking Gibbs entropy and model compression performance within deep learning iterative cycles. By treating remaining network weights as measurement vectors in a statistical mechanics context, the study establishes an inherent connection between thermodynamic efficiency and lossy compression ratios using methods like random pruning and dual tomographic compression. The work advances fundamental understanding of how randomness quantifies information retention during aggressive weight reduction techniques.
Gibbs randomness-compression proposition
arXiv:2505.23869v5 Announce Type: replace-cross Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {\it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.
Abstract:A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {\it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.
10 pages, 5 figures, 2 tables, 1 algorithm pseudocode. New references, text improvements and concept mapping for terms. Codes are available on Zenodo repository: this https URL
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2505.23869 [stat.ML]
(or arXiv:2505.23869v5 [stat.ML] for this version)
From: Mehmet A. Süzen PhD [view email] [v1] Thu, 29 May 2025 10:48:35 UTC (367 KB) [v2] Mon, 30 Jun 2025 12:42:09 UTC (438 KB) [v3] Mon, 5 Jan 2026 14:07:39 UTC (374 KB) [v4] Thu, 16 Jul 2026 14:26:46 UTC (478 KB) [v5] Thu, 23 Jul 2026 08:33:14 UTC (95 KB)
Full text under the publisher's open licence · 286 words
Related coverage
Relating Almost Perfect Condensation Conditions and the Conditioned Score
LessWrong · Jul 31, 2026
An End-to-End Hybrid Quantum--Classical Sampling Workflow for Discrete Markov Random Fields: A Reproducible Case Study
arXiv · Aug 31, 2026
HSM designs upgrade entropy sources with quantum randomness
Quantum Zeitgeist · Oct 4, 2026
Confirming Claims of Superposition and Adversarial Examples in Toy Models
LessWrong · Aug 1, 2026