aiqbench analysis
This paper tests whether JEPA-style latent prediction can replace masked language modelling for pretraining transformers on longitudinal EHRs, and finds that it consistently underperforms MLM on MIMIC-IV even though training stays stable. The result is a useful negative finding: avoiding representational collapse is not enough, and defining clinically meaningful latent targets remains the hard part. Readers working on self-supervised learning for sequential medical data should treat MLM as the stronger baseline for now.
Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
arXiv:2610.29528v1 Announce Type: new Abstract: Masked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states. Joint-Embedding Predictive Architectures (JEPAs) instead learn by predicting representations in latent space and have shown strong performance in a variety of domains. In this paper, we investigate to what extent JEPA can be adapted to EHR sequence representation. Our empirical investigation using MIMIC-IV shows that although all three of our JEPA formulations train stably without representational collapse, they consistently underperform standard MLM. We find that target size, not target heterogeneity, increases prediction error, yet reducing the target size does not recover downstream performance. This suggests that stable latent prediction alone is insufficient for competitive EHR representation learning and highlights the challenge of defining clinically meaningful latent targets for EHR.
Title:Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
Abstract:Masked language modelling (MLM) is the dominant pretraining objective for transformer encoders applied to longitudinal electronic health records (EHRs). However, reconstructing individual events may overemphasise noisy or weakly informative observations and does not explicitly encourage learning higher-level patient states. Joint-Embedding Predictive Architectures (JEPAs) instead learn by predicting representations in latent space and have shown strong performance in a variety of domains. In this paper, we investigate to what extent JEPA can be adapted to EHR sequence representation. Our empirical investigation using MIMIC-IV shows that although all three of our JEPA formulations train stably without representational collapse, they consistently underperform standard MLM. We find that target size, not target heterogeneity, increases prediction error, yet reducing the target size does not recover downstream performance. This suggests that stable latent prediction alone is insufficient for competitive EHR representation learning and highlights the challenge of defining clinically meaningful latent targets for EHR.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.29528 [cs.LG]
(or arXiv:2610.29528v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.29528 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 253 words