analiza aiqbench
This study presents WaterBERT, a domain-adapted language model trained on over 2.97 billion tokens of water treatment literature to enhance semantic representation and information extraction. The approach demonstrates that specialized continual pretraining significantly outperforms general-purpose models in downstream tasks like named entity recognition and relation extraction within this specific scientific domain. By enabling efficient processing of hundreds of thousands of abstracts, the work offers a scalable solution for mining unstructured knowledge in vertical industries without relying on expensive commercial LLMs.
Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining
arXiv:2609.26034v1 Announce Type: new Abstract: Water treatment research is expanding rapidly, but much of the knowledge acquired from this research remains scattered across unstructured literature. The field still lacks a dedicated language model that can efficiently capture water treatment-specific domain semantics for large-scale literature mining. Here, we address this by developing WaterBERT, a domain-adapted encoder model designed for semantic representation and structured information extraction from water treatment texts. WaterBERT was developed by continual pretraining on a large-scale water treatment corpus comprising about 2.97 billion tokens. Three fine-tuned models based on WaterBERT were systematically evaluated on downstream tasks, achieving the best overall performance among general-purpose and domain-specific BERT models, with F1 scores of 90.12% for multiclass treatment process classification, 79.50% for named entity recognition, and 74.04% for relation extraction. Beyond these benchmark tasks, we further demonstrated WaterBERT's advantages for large-scale literature processing. Applied to 5,144 Environmental Science & Technology articles, WaterBERT-BERTopic identified coherent, diverse, and domain-specific research topics without predefined categories. Building on WaterBERT, we processed 693,211 abstracts at substantially lower cost than commercial LLMs while retaining competitive extraction performance to construct a structured water treatment knowledge graph. The knowledge graph was then integrated with lexical and dense retrieval to develop a Water Knowledge-Enhanced Retrieval System (WaterKERS), which achieved a relevance score of 77.7, substantially outperforming text-based retrieval baselines (54.7-64.5). Through WaterBERT, this study provides a compact and scalable semantic foundation for large-scale information processing and evidence mapping in water treatment research.
Authors:Mudi Zhai (1), Ruihong Qiu (2), Qingyun Zeng (3,4), T. David Waite (1), Bing-Jie Ni (1), Haoran Duan (1,5) ((1) UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia (2) School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD 4072, Australia (3) Microsoft Copilot Studio AI, Redmond, WA 98052, United States (4) Departments of Mathematics & Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States (5) Department of Civil Engineering, The University of Hong Kong, Pokfulam, Hong Kong SAR, China)
Abstract:Water treatment research is expanding rapidly, but much of the knowledge acquired from this research remains scattered across unstructured literature. The field still lacks a dedicated language model that can efficiently capture water treatment-specific domain semantics for large-scale literature mining. Here, we address this by developing WaterBERT, a domain-adapted encoder model designed for semantic representation and structured information extraction from water treatment texts. WaterBERT was developed by continual pretraining on a large-scale water treatment corpus comprising about 2.97 billion tokens. Three fine-tuned models based on WaterBERT were systematically evaluated on downstream tasks, achieving the best overall performance among general-purpose and domain-specific BERT models, with F1 scores of 90.12% for multiclass treatment process classification, 79.50% for named entity recognition, and 74.04% for relation extraction. Beyond these benchmark tasks, we further demonstrated WaterBERT's advantages for large-scale literature processing. Applied to 5,144 Environmental Science & Technology articles, WaterBERT-BERTopic identified coherent, diverse, and domain-specific research topics without predefined categories. Building on WaterBERT, we processed 693,211 abstracts at substantially lower cost than commercial LLMs while retaining competitive extraction performance to construct a structured water treatment knowledge graph. The knowledge graph was then integrated with lexical and dense retrieval to develop a Water Knowledge-Enhanced Retrieval System (WaterKERS), which achieved a relevance score of 77.7, substantially outperforming text-based retrieval baselines (54.7-64.5). Through WaterBERT, this study provides a compact and scalable semantic foundation for large-scale information processing and evidence mapping in water treatment research.
Computation and Language (cs.CL)
arXiv:2609.26034 [cs.CL]
(or arXiv:2609.26034v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.26034 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 355 cuvinte
Subiecte conexe
Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax
LessWrong · 9 oct. 2026
Representation Editing for Multimodal Test-Time Adaptation
arXiv · 30 sept. 2026
Label-Conditioned Cross-Modal Fusion for Adult-to-Pediatric ECG Transfer via Curriculum-Gated Contrastive Alignment
arXiv · 9 iun. 2026
OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes
arXiv · 7 oct. 2026