Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

Abstract:Whole-slide pathology report generation requires models to integrate localized histomorphology, global tissue context, and diagnostically relevant semantic information, yet existing approaches typically rely on fixed pretrained visual representations and may fail to represent key diagnostic concepts. Here we present SCOUT: Semantic Context-aware mOdality fUsion Transformer, a concept-grounded multimodal framework that integrates local histological patterns, whole-slide context, and expert-curated diagnostic concepts. SCOUT maintains an evolving visual representation together with recursively updated slide- and concept-conditioned representations, allowing visual and contextual information to iteratively co-evolve across encoder depth. During decoding, separate attention pathways attend to these complementary streams before an adaptive multimodal fusion module combines them for each generated token. We evaluated SCOUT on TCGA-BRCA, HistAI, and REG-2025, comprising more than 20,900 WSI-report pairs across more than seven cancer types. Using common CONCHv1.5 features and evaluation protocols, SCOUT outperformed WSI-Caption, HistGen, and Bi-Gen, improving BLEU scores by 9.6% on average, METEOR by 11.0%, and ROUGE-L by 3.6% relative to the strongest competing method. On REG-2025, SCOUT additionally improved the Clinical Report Quality Score by 5.6%, indicating better preservation of clinically relevant report attributes beyond lexical similarity. Ablations showed complementary contributions from iterative context refinement and adaptive fusion, while learned gates provided inspectable token-level estimates of modality use. Our results suggest that progressive contextual conditioning is effective across heterogeneous pathology report generation settings and provides a flexible framework for integrating domain knowledge into vision-language models for computational pathology.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2605.01144 [cs.CV]

(or arXiv:2605.01144v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2605.01144 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.