Contributor: Vidya Murthy, WEMBA’42
To learn more about Vidya, click here.
The integration of artificial intelligence (AI), machine learning (ML), and large language models (LLMs) into healthcare has impacted many aspects of patient care and operational efficacy, but also introduced new challenges.
Regulatory Considerations
The use of AI, ML, and LLMs in healthcare marked nearly every major earnings call for the last year. But there has been much criticism, including a recent study showing half of the FDA-approved AI medical devices are not trained on real patient data. And another study showed nearly half of FDA-authorized AI tools may provide little benefit.
The Department of Justice recently issued guidance that broadly seeks to include AI in assessing compliance programs. Furthermore, the Food and Drug Administration (FDA) has issued key standards spanning a few key areas:
- Pre-market review: Medical devices using AI/ML must undergo a pre-market review that includes assessing intended use, risk level, and how AI/ML are being integrated.
- Good Machine Learning Practice (GMLP): This guidance aims to ensure safe, effective, and high-quality AI/ML application
- Real-World Evidence and Post-Market Surveillance: Given some AI/ML models evolve over time, the FDA has emphasized the importance of post-market vigilance.
While the FDA is exploring a “Predetermined Change Control Plan” (PCCP), which would allow manufacturers to make certain modifications without needing to re-submit the software for approval, that review has yet to be completed.
Globally, the challenges persist –European Union General Data Protection Regulation (EU GDPR) strives to limit personal data used in AI/ML design and require explanations of how AI-derived decisions are made to patients. EU Medical Device Regulation (MDR) strives to follow similar principles, with the European Commission having proposed an Artificial Intelligence Act to harmonize requirements around transparency, human oversight, data quality, and risk management.
Broadly, many regulators are also including ethical considerations, highlighting algorithm bias, transparency, and explainability as challenges that must be overcome prior to approval.
What does this have to do with security?
The regulatory environment for AI/ML is complex and varied, but the commonality is that to meet these requirements, security efforts must be implemented as a backbone. Trying to pursue AI/ML application development without security would be like trying to decorate the inside of a house before deciding on the house layout.
Security requirements tied to AI/ML derive from several sources in healthcare, not limited to:
- HIPAA: requires administrative, physical, and technical safeguards
- US FDA: risk assessments, monitoring, and ongoing maintenance
- EU GDPR: needs data protection by design, breach notification, and data minimization
- EU Network & Information Systems (NIS Directive): security measures and incident reporting
While the “asks” can seem overwhelming, the great thing is compliance can often be achieved by understanding industry standards, such as National Institute of Standards and Technology (NIST) Cybersecurity Framework, ISO/IEC 27001 and HITRUST Common Security Framework.
“So What?”
In 2024, water utilities, agriculture, and the power industry all faced dramatic increases in cyberattacks. But healthcare has been a “top” target for more than a decade, with 2024 including Change Healthcare, OneBlood, and CrowdStrike, with the outcome of such security lapses having a direct impact on the ability for a patient to receive care. Yet we still see rural hospitals not accessing free programs designed to help them become more cybersecure.
While you may not directly work in a hospital, the impact of security on the entire healthcare industry is directly on the bottom line. Additionally, healthcare continues to grow inorganically - data suggests S&P 500 companies are acquiring assets at a rate 4.4 times greater than they divest, up from 3.7 in the prior five years. The theme of 2024 cyberattacks was supply chain integration. Every new acquisition introduces a new supply chain that must be accounted for and secured.
If we’re ever going to get out of the catch-up game, we need to start treating security as a business imperative and prioritizing it the same way we do critical business priorities. 2025 is a chance to rewrite how we strategically position security and treat it as an enabler vs. a cost-center.
Contact Vidya at: [email protected]