Frequency-Ordered Tokenization for Better Text Compression

Abstract:We present frequency-ordered tokenization (FOT), a simple preprocessing technique that improves lossless text compression by exploiting the power-law frequency distribution of natural language (Zipf's law). FOT segments text into the vocabulary of a pretrained Byte Pair Encoding (BPE) tokenizer, renumbers tokens by frequency with a fixed table, and writes them with a short byte code before any standard compressor. Because the FOT stream is less than half the size of the text, compressors produce smaller output and run faster at the same time. On enwik8 (100 MB of Wikipedia), FOT reduces the compressed size by 7.96 percentage points (pp) for zlib, 2.24 pp for xz (LZMA), 1.96 pp for brotli and 1.44 pp for zstd, while making total compression, including the transform, 1.6x, 3.2x, 1.8x and 5.0x faster. Gains hold from 100 KB to 1 GB (enwik9) and across texts of diverse languages, and outperform the classical Word Replacing Transform. On enwik9, FOT with tuned xz sets a new state of the art among LZ-based compressors on the Large Text Compression Benchmark. The method can be implemented in under 50 lines of code.

6 pages, 4 figures, 8 tables. v2: substantially revised; all results include per-file overhead; new open-source C implementation and benchmarks. Extended version of a paper submitted to DCC 2027. Code: this https URL

Information Theory (cs.IT); Computation and Language (cs.CL)

arXiv:2602.22958 [cs.IT]

(or arXiv:2602.22958v2 [cs.IT] for this version)