GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

Title:GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

Abstract:We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two systems yield 11.6%, 16.8%, and 36.0% test MAPE. A low-bit model ladder changes ordering across runtime stacks. The P2 prefill baseline gives 18.7%, 22.2%, and 108.2% test MAPE. GGUF structure helps on all three systems, but fitted efficiencies are not universal.

27 pages: 5-page manuscript with 4 figures, followed by a 22-page supplementary results appendix. Submitted to ICASSP 2027

Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Performance (cs.PF)

arXiv:2609.14864 [cs.AI]

(or arXiv:2609.14864v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2609.14864 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.