NLM DIR Seminar Schedule

Abstract:

Large Vision Models (LVMs), built on transformer architectures, have emerged as powerful tools for general-purpose visual representation learning. This study explored the application of LVMs for medical knowledge adaptation, with a focus on automated pneumonia severity prediction from chest X-rays. Leveraging models such as ViT, DINOv2, and BiT, we investigated the roles of LVM as feature encoders under both direct extraction and self-supervised learning settings. Using a dataset of 2,599 chest X-rays labeled with mRALE scores, we evaluate multiple configurations, including Simple Siamese Network based contrastive learning, to assess how LVMs adapt to the medical domain with limited labeled data. The results show that LVMs, particularly when combined with self-supervised learning, can achieve state-of-the-art (SOTA) performance across multiple regression metrics (MSE: 23.83, 95%CI 22.67-25.00; MAE: 3.64, 95%CI 3.54-3.73; explained variance: 0.66, 95%CI 0.63-0.68; R²: 0.65, 95%CI 0.63-0.67; Spearman’s ρ: 0.80, 95%CI 0.77-0.81), when trained on a modest dataset. Notably, while self-supervised learning can enhance general feature quality, it has limited impact on improving alignment with human expert judgment. This work highlights the effectiveness and limitations of LVM-based medical AI pipelines and sets the stage for future efforts involving multi-modality integration and cross-domain adaptation. Ongoing and future work aims to extend this framework through LLM integration and multi-agent learning systems for automated, scalable medical image analysis.