Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization

Abstract:Generative medical event models use tokenized sequences of patient timelines as input, but practical guidance on the many decisions around tokenization is limited. We benchmark quantization granularity, reference-range anchoring, code--value fusion, numeric and temporal encodings, and native versus harmonized event representations from an expert-mapped common data model. Using both Llama and Qwen architectures, 156 models were trained from three initialization seeds, with each configuration following a shared training recipe for up to five epochs. We evaluated learned representations from the first 24 hours of hospitalization with linear probes to predict binary and continuous outcomes during hours 24-48. Fused tokens pairing codes with value deciles increased performance across all eight outcome families relative to the equivalent unfused tokenized input with area under the receiver operating characteristic curve (AUROC) gains of $+0.002$ to $+0.033$ and Spearman correlation gains of $+0.025$ to $+0.114$. Neither anchoring value bins to reference ranges nor increasing quantization granularity consistently improved performance, while xVal variants underperformed both discrete and soft encodings. Alternatives to explicit time tokens, such as event-order and admission-relative rotary position embeddings (RoPE), yielded higher family-mean point estimates across all eight families while reducing input length. When evaluating native input against input mapped to the Common Longitudinal Intensive Care Unit Data Format (CLIF), the CLIF pipeline used 30.8\% as many training tokens while improving performance in six of eight outcome families. These findings show tokenization and event encoding are consequential design choices when learning patient representations for downstream classification and regression.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2604.16775 [cs.LG]

(or arXiv:2604.16775v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2604.16775 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.