aiqbench analysis
This research paper investigates the statistical concentration observed when language models generate lottery numbers, revealing that LLMs fail to produce truly uniform random distributions despite their training on vast datasets. The study highlights a critical limitation in current model architectures for stochastic tasks, showing that generated outputs cluster around specific patterns rather than adhering to independent uniform sampling. For AI practitioners and researchers, this underscores the need for specialized mechanisms or post-processing steps if probabilistic generation is required for applications demanding genuine randomness.
Ask a Language Model for Lottery Numbers: Concentration in Repeated Six-of-49 Outputs
arXiv:2610.00052v1 Announce Type: cross Abstract: We evaluate six language-model configurations on requests for six distinct random integers from 1-49. Across 1,200 attempted calls using four English prompt variants, 1,184 responses yielded valid tickets. Effective diversity of number frequencies ranged from 9.9 to 18.0, compared with simulated fifth-percentile thresholds of 46.6-46.7 under independent uniform six-of-49 sampling at the corresponding sample sizes. Systems produced 8-93 distinct unordered tickets, and their modal tickets accounted for 22.5-68.0% of valid responses. Two archived Polish Lotto samples provided a physical-lottery comparison, with effective diversities of 41.1 and 41.4 at smaller sample sizes. These results demonstrate substantial concentration under the tested deployment settings. They do not identify its mechanism or establish performance under other prompts, temperatures, or tool configurations.
Abstract:We evaluate six language-model configurations on requests for six distinct random integers from 1-49. Across 1,200 attempted calls using four English prompt variants, 1,184 responses yielded valid tickets. Effective diversity of number frequencies ranged from 9.9 to 18.0, compared with simulated fifth-percentile thresholds of 46.6-46.7 under independent uniform six-of-49 sampling at the corresponding sample sizes. Systems produced 8-93 distinct unordered tickets, and their modal tickets accounted for 22.5-68.0% of valid responses. Two archived Polish Lotto samples provided a physical-lottery comparison, with effective diversities of 41.1 and 41.4 at smaller sample sizes. These results demonstrate substantial concentration under the tested deployment settings. They do not identify its mechanism or establish performance under other prompts, temperatures, or tool configurations.
6 pages, 1 figure. Data, code, and collector at this http URL (research/lotto-models)
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.00052 [cs.IR]
(or arXiv:2610.00052v1 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2610.00052 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 231 words
