UltraPIPS: Foundation Models for Ultrasound Perceptual Similarity

What changed

Researchers have introduced UltraPIPS, a new library offering Learned Perceptual Image Patch Similarity (LPIPS) metrics tailored for B-mode ultrasound imagery. Traditional LPIPS metrics often rely on backbones pretrained on natural images, but these models struggle to capture the distinct speckle patterns and acoustic-specific statistics inherent in ultrasound data. The UltraPIPS library addresses this by incorporating foundation models trained specifically on ultrasound images. The research compares the effectiveness of LPIPS metrics using backbones from natural image models, general medical image models, and ultrasound-specific foundation models across downstream tasks such as classification, segmentation, and reconstruction. Findings indicate that ultrasound backbone models show a stronger correlation with the performance of supervised models than conventional or natural image models. Furthermore, optimizing LPIPS loss with an ultrasound backbone resulted in a superior balance between reconstruction quality and image realism.

Candidate Backbones Analyzed:

• Natural Image: AlexNet, VGG-16, ViT, SwinViT, CLIP
• Radiology: RadImageNet, MedSAM, BiomedCLIP
• Ultrasound: Ultrasound Foundation Model (USFM), Texture Ultrasound Semantic Analysis (TUSA), Ultrasound-CLIP

The study utilized open-source, pretrained models and datasets. Experiments were conducted using the MONAI library for RadImageNet LPIPS calculations and the original LPIPS implementation for AlexNet and VGG-16. For ViT and SwinViT variants, features were extracted from the final layer. The LPIPS loss was computed as the L2 loss between channel-normalized feature vectors of image pairs. The research was executed on a Kubernetes cluster with NVIDIA RTX A5000 GPUs.

Why it matters for builders

The development of UltraPIPS is significant for AI builders working with medical imaging, especially in the ultrasound domain. It underscores that the choice of backbone for perceptual similarity metrics is not a trivial design decision and has a direct impact on model performance. By providing domain-specific LPIPS metrics, UltraPIPS enables developers to build more accurate and realistic ultrasound image analysis and reconstruction systems. This is particularly relevant for tasks where subtle perceptual differences are crucial for diagnosis or treatment planning.

Practical impact

Developers can now integrate the UltraPIPS library, available on GitHub, into their ultrasound imaging pipelines. This library offers a set of LPIPS metrics powered by open-source foundation models analyzed in the paper. Builders can experiment with these ultrasound-specific backbones to improve the performance of their models in tasks like image synthesis, implicit neural representations, and image reconstruction. The research suggests that using an ultrasound backbone for LPIPS loss optimization can lead to better outcomes in terms of both reconstruction fidelity and visual realism, potentially improving diagnostic accuracy and clinical utility of AI-powered ultrasound tools.

Caveats and source limits

The research presented is a preprint and has not undergone peer review. The study focuses on B-mode ultrasound and may not generalize to other ultrasound modalities or medical imaging types. While the paper analyzes several open-source foundation models, it notes a limitation to ViT and SwinViT variants due to the current landscape of available CNN-based ultrasound foundation models. The specific performance gains are demonstrated through correlation with downstream tasks and optimization of reconstruction quality, but direct, independent benchmark results comparing UltraPIPS against other state-of-the-art methods on standardized datasets are not detailed in the provided excerpt. The code is available, but details on ease of integration or specific API usage are not elaborated.