Teaching AI to speak the language of pathology

Artificial intelligence is already helping clinicians spot patterns in diagnostic images and detect signs of disease. But many of today’s pathology AI systems are built for a single task and must be rebuilt for each new application.

A study recently published in Nature Medicine describes a different approach aimed at advancing research in this field. Researchers from Microsoft Research and Paige, now part of Tempus, developed PRISM2, a pathology foundation model trained on both tissue images and language based on real pathology reports. The goal was to help AI learn from the large amount of available text data and create a system that could support future pathology research and development.

In testing, the model matched or exceeded the performance of specialized cancer-detection systems on several benchmark tasks, including prostate cancer, breast cancer and breast lymph node metastasis detection. Importantly, researchers reported these results without creating a separate model for each task.

The full PRISM2 model weights are publicly available for research use on Hugging Face here and here.

Pathology sits at the center of many cancer diagnoses. Pathologists examine tissue samples and create reports that guide treatment decisions. As healthcare generates larger volumes of data, researchers have been exploring how AI can support clinicians in making these decisions and possibly even deliver new insights.

Most pathology AI systems today are designed for a single purpose, such as detecting a particular type of cancer. PRISM2 was developed as a research effort to support a broader range of tasks through a single model that can respond to prompts and questions. Researchers say that could make it easier to build and adapt future pathology tools without starting from scratch for each new application.

The central idea behind PRISM2 is that pathology is not only a visual discipline. It is also a language-driven one.

To train the model, researchers paired pathology images with information derived from pathology reports, creating millions of question-and-answer examples that helped connect visual findings with diagnostic language. The resulting model can work with either images only or images and text, enabling users to interact with it through prompts rather than relying solely on task-specific software.

This approach differs from many earlier pathology foundation models, which focused primarily on learning visual representations from images. PRISM2 was designed from the outset to link visual patterns in tissue with the language clinicians use when making diagnoses.

Lead image created with Microsoft Copilot.