Size Matters: Foundation Model for Czech HTML documents

Abstract:Creating universal, high-quality representations of web documents in high-traffic industrial environments requires models that are both performant and economic. Existing approaches, however, often depend on large models, overlook the structural information inherent in HTML, or are constrained by short context windows, limiting their ability to process real-world web pages. We present HTML-LM, a compact foundation model with 154 million parameters that addresses these limitations through HTML-aware training and a ModernBERT-based architecture. It was trained on 100 million web documents using multiple objectives, including masked language modeling, bag-of-words prediction, and contrastive distillation from large language models. Consequently, HTML-LM sets a new state-of-the-art for classification and regression applications in the Czech Internet domain, surpassing both larger encoders and small-sized LLMs. The model is deployed in production, processing thousands of web documents per second, and released to the community under the CC BY-NC 4.0. this https URL.

Computation and Language (cs.CL)

arXiv:2609.18494 [cs.CL]

(or arXiv:2609.18494v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.18494 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

Text, Speech, and Dialogue. TSD 2026. Lecture Notes in Computer Science(), vol 16940 141-152

https://doi.org/10.1007/978-3-032-37249-9_12 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.