Learn more about how Microsoft is enhancing healthcare with data and responsible AI. Read the latest Microsoft Cloud for Healthcare announcements. 

Existing language models have revolutionized how we interact and use powerful AI models for text-based use cases in healthcare. But the practice of modern medicine is chiefly multimodal. Effectively assessing the complete picture of patient health requires moving beyond medical text comprehension to sophisticated AI models capable of integrating and analyzing diverse data sources across modalities such as medical imaging, genomics, clinical records, and more.

The creation of comprehensive multimodal models has traditionally been hindered by the need for large-scale, integrated datasets and the significant computational power needed to train these models. These barriers have limited the ability of many healthcare organizations to fully leverage AI.

Microsoft Cloud for Healthcare helps to bridge this gap and accelerate AI development. We’re announcing the launch of health AI models, a collection of cutting-edge multimodal medical imaging foundation models available in the Microsoft Azure AI model catalog. Developed in collaboration with Microsoft Research and strategic partners, these AI models are specifically designed for healthcare organizations to test, fine-tune, and build AI solutions tailored to their specific needs, all while minimizing the extensive compute and data requirements typically associated with building multimodal models from scratch. With healthcare AI models, health professionals have the tools they need to explore the full potential of AI to transform patient care.

Healthcare AI models include:

• MedImageInsight: An embedding model enables sophisticated image analysis, including classification and similarity search in medical imaging. Healthcare organizations and researchers can use the model embeddings and build adapters for their specific tasks, streamlining workflows in radiology, pathology, ophthalmology, dermatology, and other modalities. For example, researchers can explore how the model can be used to build tools to automatically route imaging scans to specialists, or flag potential abnormalities for further review, enabling improved efficiency and patient outcomes.
• MedImageParse: Designed for precise image segmentation, this model covers various imaging modalities, including x-rays, CTs, MRIs, ultrasounds, dermatology images, and pathology slides. It can be fine-tuned for specific applications such as tumor segmentation or organ delineation, allowing developers to test and validate the ability to leverage AI for highly targeted cancer and other disease detection, diagnostics, and treatment planning.
• CXRReportGen: Chest x-rays are the most common radiology procedure globally. They’re crucial because they help doctors diagnose a wide range of conditions—from lung infections to heart problems. These images are often the first step in detecting health issues that affect millions of people. By incorporating current and prior images, along with key patient information, this multimodal AI model generates detailed, structured reports from chest x-rays, highlighting AI-generated findings directly on the images to align with human-in-the-loop workflows. Researchers can test this capability and the potential to accelerate turnaround times while enhancing the diagnostic precision of radiologists. This model has demonstrated exceptional performance on the industry standard MIMIC-CXR benchmark (source paper).

These foundational models can accelerate the arrival of groundbreaking AI models that bring intelligent workflows, efficient report generation, and advanced view identification and segmentation to the radiologist experience. In addition to supporting report accuracy, AI can help advance patient care by unlocking new insights from radiology and pathology and genomics, accelerating the discovery of new treatments for disease, and predicting outcomes and optimal treatment plans.

Resources are no longer a barrier to innovation

With so many demands on healthcare and life sciences organizations, it’s challenging to dedicate time, resources, and budget to experiment with AI. Healthcare AI models feature open source, pretrained models that represent some of the highest level of performance currently achievable on public benchmarks.

In aggregate, the healthcare AI models and others in our catalog of multimodal medical foundation models span a wide range of modalities and a growing catalog of competencies, enabling the testing and validation of a wide range of use cases, including:

• Using an image embedding model to search for similar images or facilitate detection of anomalies that could indicate potential data issues or system errors (Fig. 2: Image embedding).
• Building an adapter to the embedding model for a specific task. (Fig. 3: Adapter to specific task)
• Fine-tuning pretrained unimodal models to create a narrow model. (Fig. 4: Fine-tuning for a specific task)
• Integrating language models to enable the extraction of insights across modalities and enhance the interpretability of multimodal data. (Fig. 5: Adapter to general reasoner)
• Connecting different data modalities for a more comprehensive, holistic view of data that derives new insights and enables the discovery of previously hidden correlations and patterns.

With the flexibility and breadth of models, individual unimodal health models can be used independently, connected to different modalities, or further combined with advanced general reasoning models like GPT-4o and Phi to create powerful multimodal models without the need for massive integrated datasets from the outset. Azure AI Studio and healthcare AI models complement the healthcare data solutions available in Microsoft Fabric, creating a unified environment for comprehensive analysis and vital patient insights.

Created by a collaborative network of partners

Our ecosystem of partners dedicated to advancing the industry’s use of AI made healthcare AI models possible. Paige, Providence Healthcare, Nvidia, and M42 contributed foundational models to the catalog, spanning pathology, 3D medical imaging, biomedical research, and medical knowledge sharing. Developed under a core set of shared AI principles, these models provide a powerful starting point for organizations as they launch their own AI projects, while embedding responsible practices across the industry. Microsoft is committed to responsibly scaling AI and to listen, learn, and improve our tools. We work with organizations to help them harness the data to build the predictive and analytical power required for their own competitive advantage.

The open access to AI models on the catalog and modular approach allows healthcare organizations to customize solutions, maintain control over their data, and build trust through shared development and oversight. This approach aligns with our commitment to responsible AI, ensuring our technologies meet ethical standards and earn the trust of the medical community.

The catalog’s ongoing evolution will be a collaborative effort—not just among those providing foundational models, but also with the support of customers and partners that are building on these models to develop their own research or clinical systems.

Microsoft is committed to fostering transparency and community involvement within an ecosystem that empowers partners, developers, and researchers to push the boundaries of what is possible in healthcare and empower healthcare and life sciences organizations to achieve more. It’s not just about building models; it’s about unlocking new insights, accelerating innovation, and ultimately improving patient outcomes on a global scale, from pioneering cutting-edge pharmaceutical research to delivering life-changing medical care.

Innovation in action

Several customers are already taking advantage of the possibilities unlocked by healthcare AI models.

Mass General Brigham and University of Wisconsin are targeting advanced report generation from medical imaging analysis. With ever-increasing imaging volumes colliding with the ongoing combination of radiologist burnout and shortages, a state-of-the-art medical imaging model can be used to build an application that can transform a medical image into a draft note. Projects like these can transform the efficiency of core healthcare workflows, supporting better outcomes for patients while helping clinicians focus on the hands-on components of their roles.

Grounded report generation from medical images is a new frontier. Our shared collaboration brings diverse expertise to developing, testing, and validating new models. We are working to identify and overcome the challenges of how models can be integrated into real clinical systems and workflows so that a pathway exists for these capabilities to have the potential to impact real patient care in the future.

—Richard Bruce MD PhD, radiology Vice Chair of Informatics, University of Wisconsin-Madison

In life sciences, Paige is working to combine radiology, pathology, and genomic insights for a more comprehensive approach to disease diagnosis, aimed at accelerating the discovery of new treatments. AI has a key role to play throughout the healthcare continuum, and advances made in our understanding of risks, diseases, and treatments will be instrumental for improving downstream patient care.

The collaboration with Microsoft has enabled Paige to unlock insights from millions of digitized pathology slides, clinical reports, and genomic data, to gain a more holistic understanding of cancer. Together, we are pioneering frontier multi-modal AI models that have the potential to accelerate and redefine cancer detection, diagnosis, and treatment. We are thrilled to continue to lead the charge and shape the future of precision oncology.

—Razik Yousfi, Chief Executive Officer & Chief Technology Officer of Paige

And it’s not just human health that healthcare AI models are supporting; Mars PETCARE is exploring use cases in veterinary medicine, such as data evaluation for radiology and pathology teams. Treating pets is every bit as complicated as treating humans, so this work just goes to show the platform’s versatility—each of these models can be turned to a novel application with the right approach.

Our strategic partnership with Microsoft represents a significant leap forward in veterinary diagnostics. As early adopters of AI in digital pathology and radiology, we’ve seen firsthand how this technology can transform animal care. By combining our veterinary expertise with Microsoft’s frontier AI models, we’re not just advancing diagnostics, we’re creating a better world for pets. This collaboration will accelerate our AI R&D [research and development] efforts, empowering veterinarians with more accurate and efficient tools. Together, we’re setting new standards in veterinary medicine and reinforcing our commitment to innovation in animal health.

—Jerry Martin, Vice President, Research & Development, Mars Science & Diagnostics

Sectra is exploring how image and text embeddings from foundational models can be leveraged to transform workflow tasks in radiology. Traditionally managed through static configurations, these tasks are now being revamped to adapt to the diverse nature of healthcare data using generative AI.

—Fredrik Häll

Topcon Healthcare is building a multimodal and three-dimensional ophthalmic imaging Foundation Model (FM) to phenotype healthy populations by leveraging data collected from large population-based screening environments. This FM facilitates exploration of biomarkers in the eye that are early indicators of eye and systemic diseases.

—Mary Durbin, Vice President of Clinical Science, Topcon Healthcare

We are excited to offer Med42, our leading clinical LLM, through Azure AI Studio. With Med42, we are harnessing the power of AI to impactfully disrupt traditional healthcare systems and deliver value for clinicians, scientists, and patients. With advancements like our Med42 suite of healthcare foundation models to MEDIC, our comprehensive clinical evaluation framework for LLMs, M42 is advancing global innovation in healthcare.

—Dr. Ronnie Rajan, Associate Director, AI & Applied Science, Med42

The development of foundational AI models in pathology and medical imaging is expected to drive significant advancements in cancer research and diagnostics. These models can complement human expertise by providing insights beyond traditional visual interpretation, and as we move toward a more integrated, multimodal approach, will reshape the future of medicine.

—Carlo Bifulco, MD, Chief Medical Officer, Providence Genomics and a co-author of the Prov-GigaPath study

Microsoft Cloud for Healthcare is helping your organization shape a healthier future with data and AI

We’re excited to strengthen our data and AI investments through the Microsoft Cloud for Healthcare. Our healthcare solutions are built on a foundation of trust and Microsoft’s Responsible AI principles. Through these innovations, we’re making it easier for our partners and customers to create connected experiences at every point of care, empower their healthcare workforce, and unlock the value from their data using data standards that are important to the healthcare industry.

Learn more about AI with healthcare

• Join us at HLTH2024, Booth #4004 to see healthcare AI models in action.
• Visit Microsoft Azure AI Studio to learn more about healthcare AI models and explore how its foundational models could help you transform the way your organization uses AI.
• Connect and learn about Microsoft Research’s advancements in AI and language models
• Learn more about healthcare data solutions in Microsoft Fabric.
• Watch the AI for Business Transformation: Multimodal Models interview.

Medical device disclaimer: Microsoft products and services (1) are not designed, intended or made available as a medical device, and (2) are not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used to replace or as a substitute for professional medical advice, diagnosis, treatment, or judgment. Customers/partners are responsible for ensuring solutions comply with applicable laws and regulations.

Generative AI does not always provide accurate or complete information. AI outputs do not reflect the opinions of Microsoft. Customers/partners will need to thoroughly test and evaluate whether an AI tool is fit for the intended use and identify and mitigate any risks to end users associated with its use. Customers/partners should thoroughly review the product documentation for each tool.

The post Unlocking next-generation AI capabilities with healthcare AI models appeared first on Microsoft Azure Blog.