Medical Imaging

Project Overview

This project focuses on the application of image classification techniques, a cornerstone of computer vision, to the domain of medical imaging. Recognizing the presence of a lesion in a radiograph is a fundamental step, but understanding its precise location is crucial for accurate diagnosis, treatment planning, and patient management. This project aims to achieve both lesion detection and localization by leveraging Gradient-weighted Class Activation Mapping (Grad-CAM). This technique is great for cases where you have image level labels, but not localized labels. This is a common scenario. A great example is the Stanford chest xray dataset which contains image level labels on more than 200,000 images.

Project Scope

The primary goal is to develop a robust system that not only identifies lesions in medical images but also accurately pinpoints their locations, enhancing the diagnostic process’s efficiency and accuracy. This system is intended to support radiologists and medical practitioners by providing an additional layer of analysis to assist in their evaluations. It is important to note that the aim is to build a new tool for radiologist, rather than trying to automate their role as it encompasses far more than simply identifying lesions on images.

Technical Development

Training the model requires a dataset of medical images with labels indicating the presence of various abnormalities. For this we used the Stanford chest xray dataset.

Grad-CAM allows for the visualization and localization of what the model is “focusing on” when it predicts a specific class (e.g., pneumonia present) by analyzing the gradients of the final convolutional layer of the model with respect to the class prediction. This technique does not require explicit bounding boxes or location labels, making it particularly useful for our application and our dataset.

During model training we tested with both transformer-based and convolutional neural networks (CNNs).

User Interface and Experience

Currently the user interface is built using streamlit. This provides a nice set of tools like adjustable opacity on the heatmap overlay. However this would be best built into the practitioners viewing platform. This would allow the greatest ease of use and reinforce it as yet another support tool.

Other uses of Grad-CAM

Grad-CAM can also be used as a valuable tool to detect subtle forms of data leakage or other training issues. One example I found was early on in training the model on veterinary radiology images the model was performing very well on both the training and test sets. When I applied grad-CAM I noticed it was highlighting the corner of the image where on some of the images was a subtle watermark. I learned that this watermark was generated by the viewing software after the radiologist took a measurement (e.g. measuring the size of a pulmonary nodule). Therefore the model learned to look for whether or not the watermark was present. This leads to good model performance, but not for the right reason, and of course would not be useful in practice. The solution was to black out the corner of all images.