Deep learning is revolutionizing the field of medical imaging, offering transformative potential that extends across various facets of healthcare. From enhancing diagnostic accuracy to optimizing clinical workflows and improving patient outcomes: healthcare is teeming with AI-based solutions.
The development of MRI-based radiation free bone imaging using BoneMRI is a great example of technology that would not have been as successful without contemporary deep learning. These innovations are accelerated by a growing community of committed researchers in medical imaging who keep adapting and refining cutting-edge techniques from the broader AI community to address the unique challenges posed by medical imaging applications.
We will discuss the latest trends in deep learning, which shaped technologies such as BoneMRI and similar front-running developments, will drive future innovations, their needs and potential for clinical practice.
Foundation Models
A “foundation model” is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks. (source)
There is a growing focus on foundation models. This approach involves pre-training a ‘general’ model on a large and diverse dataset. The pre-training task can be something different than the downstream task the model will eventually be used for. In order to obtain as large and as diverse a dataset as possible it is common to pre-train with a self-supervised training objective, so as to not require any paired data or annotations. When the foundation model has been pre-trained it can be fine-tuned on more specialized medical imaging tasks, possibly on a smaller dataset and with a supervised training objective that requires paired data and/or annotations. The rationale behind pre-training a large foundation model first is to build robust models that can generalize well across various tasks.
One such foundation model is the Segment Anything Model (SAM). It was designed for image segmentation tasks in natural 2D RGB images (photos scraped from the internet) and it works amazingly, but it hasn’t really been built to deal with 3D medical imaging modalities. Adapting 2D-oriented models to effectively handle 3D medical images is a significant challenge. Researchers are exploring novel architectures and methods to bridge the gap, but the tricks that have to be applied can feel awkward and ad-hoc. One notable adaptation that has been done well is Segment Any Bone Model. Fine-tuning SAM works better than training from scratch, because SAM has been pre-trained on such an enormous dataset that it has learned enough about segmentation in general to be useful for medical image segmentation. This demonstrates a great potential for foundation models; imagine a foundation model that has been built with 3D medical imaging in mind!
Images are generated with AI
Pre-training a foundation model is typically done with Self-Supervised Learning: Techniques such as contrastive learning or masked image modeling allow models to learn useful representations from unlabeled data (data without human annotated or independent ground truth) by predicting parts of the image from other parts. This approach is particularly useful in medical imaging, where labeled data is scarce but where unlabeled data is more abundant.
Fine-tuning a foundation model is a form of Transfer Learning: This involves taking a pre-trained model and fine-tuning it on a smaller, domain-specific dataset. Deep Learning practitioners can choose to update all parameters, few parameters, or somewhere in between. If too many parameters are updated, the model might forget what it learned in the pre-training stage, risking a phenomenon called catastrophic forgetting. If too few parameters are updated, the model might not learn well on the fine-tuning task. Finding the right balance is challenging and can vary per situation.
Finding the right balance is called Parameter-efficient Fine-tuning: Techniques such as Adapter modules, LoRA (Low-Rank Adaptation), and BitFit enable fine-tuning by only updating a small subset of the model’s parameters, thereby striking the optimal balance between learning the new task and maintaining previously learned skills – and reducing computational cost as a bonus.
As we started this paragraph about foundation models, this won’t come as a surprise to anyone by now: Deep Learning models require a lot of data. In medical imaging, one of the most pressing issues is the scarcity of high-quality supervised data. Unlike natural image datasets, which are often abundant and readily accessible, medical imaging data is more limited due to several factors:
- The acquisition of medical images requires specialized equipment and procedures.
- Annotation of these images have to be done by medical professionals, whose time is valuable and scarce.
- Data protection regulations, such as HIPAA and GDPR – as important as patient privacy is – limit the availability of comprehensive datasets. Even when data can be shared, ensuring data security and de-identification adds another layer of complexity.
Additionally, the advent of transformers in Deep Learning has exacerbated the hunger for Big Data. Transformers require significantly larger amounts of data compared to traditional models like convolutional neural networks (CNNs) in order to be trained effectively. This need for vast amounts of data highlights the challenge of applying these state-of-the-art techniques in medical imaging.
Synthetic Data
In search of these challenging large amounts of data, especially in the medical domain, scientists have been looking for alternatives. Can we generate large amounts of example data realistically using only a limited subset? How do we account for less commonly observed phenomena?
Synthesizing data for rare or less common pathologies has emerged as a key focus in addressing data imbalances in medical imaging datasets. Diffusion models are particularly effective at generating high-quality synthetic images of rare conditions.
Recent work has demonstrated the use of diffusion models to generate synthetic images of rare brain lesions and vascular anomalies. These are probabilistic generative models that learn to generate data by gradually reversing a diffusion process that corrupts data into noise, allowing them to generate high-quality samples by iteratively denoising random noiseBy providing additional training examples for less common pathologies, generated samples by diffusion models could enhance the ability of downstream machine learning algorithms to accurately detect and classify these conditions. This approach not only helps bridge the gap caused by data scarcity but also enables models to generalize better across various medical scenarios, including those involving rare diseases. As an added bonus, synthetic data is free from any privacy concerns, because the patient in the image doesn’t really exist.
Enhancing data
Lastly, enhancing existing medical imaging data through denoising, artifact reduction, and super-resolution is becoming increasingly important. Target data can be enhanced during training, to train the model to output clearer images, or input data can be enhanced during production to remove noise and distractors for the downstream model.
Denoising addresses noise in medical images, which can obscure important details necessary for accurate assessments. Diffusion models are not only a powerful tool in image synthesis, but also in image denoising. For example, in retinal imaging, diffusion models can significantly enhance image clarity, improving the detection of subtle abnormalities like microaneurysms or hemorrhages without additional exposure to harmful light.
Artifact reduction is vital for ensuring the diagnostic utility of medical images, as artifacts can obscure critical features and lead to misinterpretations. Artifacts can result from patient movement, equipment malfunctions, or intrinsic limitations of imaging modalities, causing noise, blurring, or distortions. Super-resolution techniques aim to enhance low-resolution images into high-resolution images, offering improved spatial resolution and continuity. This enhancement is especially valuable in modalities like MRI, where data is sometimes acquired in relatively thick 2D slices. Super-resolution should be treated with care, though. A superresolution model can never know about structures that are smaller than the resolution of the original scan, it can only make a well-informed ‘guess’ based on the large amount of data it has seen in training. So care should be taken during training that no information is hallucinated by the model, especially when applying within the medical domain. The goal of a super-resolution model is to more clearly represent the information that is present in the low-resolution scan and to ‘fill in the blanks’ only where appropriate.
Relevance to Clinical Practice
Innovations are great and all, but let’s consider what really matters: ‘does it improve patient care?’. More important than newer complex methods and improved scores on a scientific common benchmark is the ability of algorithms to handle real-world data, mitigate biases, and adapt to data drift over time. Should we adopt all of these exciting novel technologies based on the sometimes rather theoretical evidence that is provided?
Performance metrics such as Dice coefficients, Hausdorff distances, accuracy, precision, recall, and AU-ROC are often used for assessing model effectiveness. It’s great to have a quantitative yardstick to guide algorithm improvements, but we should remind ourselves that they’re a mere proxy to what we really want to optimize. Admittedly, ‘what we really want to optimize’ is hard to define and thus hard to optimize for with math and engineering. Therefore, easier-to-measure proxies are used, but we should remind ourselves that they’re not the end goal. That’s what we constantly need to remember ourselves when improving our BoneMRI models: although we enjoy the way towards the goal, it is only the end goal itself that truly matters.
In general, experts caution that despite technological advancements, there is often a disconnect between improved research performance metrics and added practical utility. Even when a model provides demonstrable improvement it is sometimes still not taken up into clinical practice because the workflow is less convenient, the benefits don’t outweigh the risks or costs, or because the metric with which improvement is observed turned out not sufficiently relevant.
Therefore, development of new AI tools in medical imaging should always keep clinical benefit in mind, with comprehensive analysis of clinical risk-benefit, workflow and cost effectiveness. Clinicians and patients, as the end users of these tools, should be central to the development process. Their feedback and requirements are crucial for ensuring that innovations lead to meaningful improvements in healthcare delivery. By focusing on real-world applicability and addressing practical challenges, researchers can ensure that deep learning solutions in medical imaging provide significant benefits to patients and healthcare providers alike.
Conclusion
The integration of deep learning into medical imaging is advancing rapidly, with innovations such as radiation free bone imaging using BoneMRI addressing unmet clinical needs, and providing safer and more effective solutions. These exciting trends are shaping the future of diagnostics, treatment and patient care. By leveraging pre-trained models, synthesizing data of rare conditions, and enhancing image quality, researchers are overcoming the limitations posed by data acquisition costs, privacy and information security concerns, and pushing the boundaries of what’s possible with the data that’s available. However, the real value of these innovations are measured by the (admittedly harder-to-measure) improvement to patient care. As we move forward, a balanced approach that merges cutting-edge technology with practical, patient-centered solutions will be crucial in realizing the full potential of deep learning in medical imaging.