Predetermined Change Control Plan: upgrading algorithms for customers

Predetermined Change Control Plan: upgrading algorithms for customers

By: David Sparks

BoneMRI algorithms 

Over the past few years MRIguidance has continuously worked on its BoneMRI product, improving the core of its medical device — the algorithms that create synthetic CT images out of MRI data. In previous blog posts it was explained how the Algorithm Team uses state-of-the-art deep learning techniques to keep improving the technology, and about the key to its success: testing. MRIguidance has created an extensive and robust training and testing workflow that allows for continuous model improvements. By setting model benchmarks using a number of different metrics, the performance can be tracked, and areas where they fall short and need fine-tuning can be identified. This iterative process of testing is at the heart of building a trustworthy deep learning model.

Regulatory clearance

To get these rigorously tested algorithms onto the market, an equally rigorous regulatory pathway must be completed, as detailed in our previous blog post about regulatory compliance. To access the US market, MRIguidance goes through a premarket submission process, and in the case of BoneMRI, a premarket notification 510(k) specifically. MRIguidance received the first FDA 510(k) clearance in 2021 and a number of additional 510(k) clearances over the past years.

The process of getting market clearance is a stringent and long process in which a manufacturer needs to demonstrate that the safety and efficacy of the device is substantially equivalent to a comparable device that is already legally approved. The performance and safety data included in a premarket submission is reviewed by experts from the FDA. When they are satisfied with the information in the submission, the FDA can decide to ‘clear’ the device to be marketed in the US.

Before the BoneMRI product is submitted for market clearance, the algorithms within are ‘locked’. They do not change after the final testing has taken place. These ‘locked’ algorithms are reviewed throughout the 510(k) premarket submission process. After clearance to the US market has been obtained, this cleared device with the ‘locked’ algorithms is the device that can be marketed. The manufacturer can not change the device that is cleared for the market (with the exception of changes that do not impact the safety and performance of the device). This means that if MRIguidance makes any improvements to the algorithms, a new premarket submission process needs to be started.

The impact of starting such a process again is big. Any improvements to the BoneMRI product would need to go through a long regulatory process before they can be deployed to the end-users, delaying efforts to bring cutting-edge technology to those patients that can benefit from it. The FDA has stated in the past that “the traditional paradigm of medical device regulation was not designed for adaptive Artificial Intelligence/Machine Learning (AI/ML) technologies” and that “The highly iterative, autonomous, and adaptive nature of these tools requires a new, total product lifecycle (TPLC) regulatory approach that facilitates a rapid cycle of product improvement”. Over the last years the FDA has worked on a new regulatory approach to increase patients’ access to safe and effective AI/ML-enabled devices.

Predetermined Change Control Plan

In 2023 the FDA published guiding principles and a draft guidance on the use of a Predetermined Change Control Plan (PCCP) which aims to “provide a forward-thinking approach to promote the development of safe and effective medical devices that use ML models”. A PCCP is a document that describes what changes a manufacturer plans to make to an already cleared algorithm and how these changes will be assessed. By describing future algorithm changes and the validation method of those changes you are essentially ‘pre-validating’ these changes. The FDA will review the content and scope of the PCCP within a regulatory submission and can give a manufacturer permission to implement the described changes without the need for a new regulatory submission — avoiding the regulatory hurdles associated with new premarket submissions.

There are three sections in a PCCP:

  • a description of the planned future changes;
  • a method or protocol to implement and validate those changes; and
  • an impact assessment of the changes.

This sounds straightforward, but in reality there is an enormous amount of effort that goes into the creation of such a PCCP. This regulatory approach is underpinned by good machine learning practices that need to be integrated throughout the development process of a company. To establish and maintain such practices is one of the key focus points of the Algorithm Team at MRIguidance. The team has spent years developing well-documented and robust processes to handle everything from data management and (re)-training methods to performance evaluation. By using a well-defined and consistent process for the algorithm improvements, they are able to build high performing and safe deep learning models time and again.

Algorithm improvements

The type of algorithm improvements that can be implemented without new regulatory submission is limited, even under a PCCP. For the BoneMRI product there are a number of improvements that fit the scope of a PCCP and benefit the end-users:

  1. Improve model performance with additional training data. Adding additional (real-world) data to the training set of the model can increase the accuracy, robustness, and performance for challenging cases such as rare pathologies or artifacts.
  2. Validation of additional scanner support. Expand the performance evaluation of the model to validate the BoneMRI product for a new MRI vendor or field strength.

These algorithm improvements are incorporated into the BoneMRI product using the extensive good machine learning practices that MRIguidance has implemented. Using a robust training and testing framework (mentioned in a previous blog post) it is possible to build trustworthy deep learning models. By adding this to a Predetermined Changes Control Plan, MRIguidance can include improvements to the BoneMRI product much more quickly, ultimately improving user and patient experience.

FDA 510(k) clearance

The latest BoneMRI product recently received 510(k) clearance for the US market. As part of this premarket submission process a Predetermined Change Control Plan has been included, with the planned future improvements detailed above. The FDA has reviewed this PCCP and has granted MRIguidance approval to implement such changes under the PCCP.

Within the complex medical device world it is important for regulators such as the FDA and manufacturers to continue innovation. With this latest approval of the PCCP, MRIguidance can keep providing its end-users high performing, state-of-the-art technology that will ultimately help patients benefit from it the most.