Gibbs randomness-compression proposition

arXiv:2505.23869v5 Announce Type: replace-cross Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {\it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.

Abstract:A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {\it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.

10 pages, 5 figures, 2 tables, 1 algorithm pseudocode. New references, text improvements and concept mapping for terms. Codes are available on Zenodo repository: this https URL

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2505.23869 [stat.ML]

(or arXiv:2505.23869v5 [stat.ML] for this version)

From: Mehmet A. Süzen PhD [view email] [v1] Thu, 29 May 2025 10:48:35 UTC (367 KB) [v2] Mon, 30 Jun 2025 12:42:09 UTC (438 KB) [v3] Mon, 5 Jan 2026 14:07:39 UTC (374 KB) [v4] Thu, 16 Jul 2026 14:26:46 UTC (478 KB) [v5] Thu, 23 Jul 2026 08:33:14 UTC (95 KB)

analiza aiqbench

Gibbs randomness-compression proposition

Subiecte conexe