AI Governance for Healthcare: HIPAA, Clinical AI, and What Regulated Organizations Actually Need

AI governance in healthcare is not the same as HIPAA compliance, and treating them as interchangeable is one of the most expensive mistakes a health system or digital health company can make. HIPAA governs how protected health information is handled, stored, and transmitted. AI governance governs whether AI systems make decisions that are accurate, accountable, fair, and defensible under an entirely different set of frameworks that HIPAA was never written to address.

Healthcare organizations deploying AI in 2026 face overlapping regulatory requirements from at least four distinct sources: HIPAA, FDA device classification rules, ONC clinical decision support guidance, and the EU AI Act for any organization with European exposure. Each layer asks different questions about different risk surfaces, and no single compliance program covers all four.

This post explains what each layer actually requires, where they overlap and where they do not, and what a healthcare AI governance program needs to cover to hold up under scrutiny from regulators, enterprise customers, and health system risk committees.

Why "HIPAA-compliant AI" is incomplete

HIPAA compliance for an AI system means the system handles PHI correctly: the vendor has signed a Business Associate Agreement, data is encrypted in transit and at rest, access controls limit who can see patient data, and audit logs capture who accessed what and when. Those are necessary conditions for deploying AI in a healthcare environment. They are not sufficient.

A predictive readmission model that handles PHI correctly but performs significantly worse for patients from certain demographic groups is HIPAA compliant. A clinical decision support tool that routes patient messages correctly but hallucinates medication dosages under certain input conditions is HIPAA compliant. An AI-powered prior authorization system that makes consequential coverage decisions without a documented audit trail of how those decisions were reached is HIPAA-compliant.

None of those systems are well governed. The governance gap is not in how PHI is handled. It is in how the AI's decisions are made, tested, monitored, and accounted for. That gap requires frameworks that HIPAA was not designed to address.

For a detailed look at what HIPAA-compliant AI deployment actually requires at the infrastructure layer, see HIPAA-Compliant AI: What Healthcare Organizations Actually Need to Do. This post covers what comes after that baseline.

The four regulatory layers healthcare AI organizations face

Layer 1: HIPAA Privacy and Security Rules

HIPAA applies to any AI system that touches PHI, which in a healthcare setting is most of them. The practical requirements for AI vendors: a signed BAA, SOC 2 Type II certification (or equivalent), encryption at rest and in transit, access logging, data retention and deletion policies, and a breach notification procedure.

What HIPAA does not require: evidence that the AI model is accurate, documentation of training data, bias testing, human oversight procedures, or any accountability structure for the AI's decisions. The regulation is about data protection, not AI quality.

Layer 2: FDA AI/ML device classification

For clinical AI systems that meet the definition of Software as a Medical Device (SaMD), FDA oversight applies in addition to HIPAA. The FDA's January 2025 draft guidance on AI-enabled device software functions requires manufacturers to address lifecycle management: how the model is tested before deployment, how performance is monitored after deployment, and what triggers a new premarket submission when the model changes.

The FDA's framework distinguishes between locked algorithms (fixed after deployment) and adaptive algorithms (continuing to learn from real-world data). Adaptive algorithms face higher scrutiny because their behavior can change after the initial regulatory review.

The key practical question for health systems deploying AI: is the clinical AI tool you are using classified as a medical device? If your vendor's AI system is making or informing clinical decisions (diagnosis, triage, treatment recommendation, risk stratification), there is a meaningful probability that it falls under FDA SaMD jurisdiction. If the vendor does not have a clear answer to whether their product is FDA-regulated, that is a governance flag before the contract is signed.

AI tools used for administrative, operational, or revenue cycle purposes generally do not trigger SaMD classification. Clinical decision support that requires clinician interpretation before action may qualify for an exemption. Direct-to-patient AI that influences a clinical decision is more likely to require FDA clearance.

Layer 3: ONC clinical decision support transparency requirements

The Office of the National Coordinator for Health Information Technology's HTI-1 rule, finalized in January 2024, established transparency requirements for certain types of clinical decision support delivered through certified EHR technology. Under these requirements, healthcare organizations and their vendors must be able to describe the source, logic, and evidence base for clinical decision support tools.

The practical implication: if you are deploying AI-based clinical decision support inside an EHR environment, you need to be able to answer when a clinician asks how the recommendation was generated. A black-box model that produces recommendations without a documented basis for its logic is increasingly difficult to defend under ONC requirements, regardless of its clinical accuracy.

Interoperability also factors in here. The FHIR standard is central to how clinical data flows between systems, and AI tools that consume or produce clinical data need to operate within that framework. An AI governance program for a health system should include a review of how AI tools integrate with the clinical data environment and whether those integrations meet ONC's interoperability standards.

Layer 4: EU AI Act exposure for US healthcare organizations

The EU AI Act, which began phased enforcement in 2024, applies to AI systems used in or affecting EU individuals, regardless of where the vendor or deploying organization is headquartered. For US-based health systems and digital health companies with any EU patient population, clinical partnerships, or regulatory submissions to European bodies, the EU AI Act creates direct obligations.

The AI Act classifies most clinical AI as high-risk: AI used for medical diagnosis, triage, clinical decision support, and patient risk assessment falls into this category. High-risk AI requires: a documented risk management system throughout the AI lifecycle; data governance documentation covering training and testing datasets; technical documentation of the AI system's design; logging sufficient for post-hoc audit; transparency to users about AI involvement in decisions; human oversight measures; and accuracy and robustness testing before deployment.

These requirements go significantly beyond what HIPAA requires. A health system that has done thorough HIPAA compliance work but has not addressed EU AI Act requirements is not compliant for its EU-exposed AI deployments.

What healthcare AI governance actually covers

A governance program that addresses all four layers is not a compliance checklist. It is a structured framework that assigns organizational accountability, defines policies, implements monitoring, and produces the evidence that regulators and enterprise customers ask for. The six dimensions that a complete healthcare AI governance program needs to address:

AI inventory and risk classification. You cannot govern what you have not cataloged. Most health systems and digital health companies deploying AI in 2026 do not have a complete inventory of the AI systems running across their environment, including the AI capabilities embedded in EHR platforms, revenue cycle tools, patient communication systems, and third-party clinical applications. The first step is a systematic inventory that classifies each AI system by risk level, regulatory exposure, and data sensitivity.

Governance accountability. Each AI system needs a named owner who is accountable for its performance, its regulatory status, and its governance. A clinical AI tool with no named organizational owner is a governance gap regardless of how good the underlying model is. The accountability structure needs to be documented and should include escalation paths for when the AI system behaves unexpectedly.

Data governance for AI. Healthcare AI systems are trained and validated on clinical data. The governance program needs to document what data was used to train each model, whether that data was representative of the patient population the model will serve, what data quality controls were in place, and how data access for AI purposes is controlled within the organization. For FDA-regulated devices, this documentation is a regulatory requirement. For all clinical AI, it is a governance best practice that directly affects model performance.

Output quality and human oversight. A clinical AI tool deployed without post-deployment monitoring is not governed, regardless of how well it performed during validation. Healthcare AI governance requires ongoing monitoring of model performance against clinical outcomes, a process for detecting model drift (the degradation of model performance over time as the real-world data distribution shifts from the training distribution), and defined human oversight procedures for high-stakes AI decisions that regulations or organizational policy require a clinician to review before action is taken.

Third-party AI risk. The majority of AI systems a health system deploys are purchased from vendors, not built internally. Governance extends to vendor AI: the organization needs to assess each AI vendor's governance practices, review its model documentation, understand its FDA regulatory status, and ensure the vendor relationship is structured with appropriate contractual protections. A health system with robust internal governance but that purchases ungoverned AI from third-party vendors has a gap.

Incident response for AI. When a clinical AI system behaves unexpectedly, who is notified? What is the escalation path? When is the system taken offline versus when it is monitored more closely? A healthcare AI governance program needs a defined incident response procedure for AI failures that is integrated with the organization's existing clinical safety and IT incident response processes.

The financial services parallel

Organizations looking at healthcare AI governance sometimes ask how it compares to AI governance in financial services, the other heavily regulated sector facing similar questions. The comparison is useful, and the differences matter.

Both sectors face regulatory pressure on AI transparency and accountability. Both have enterprise customer expectations that include AI governance attestations as a procurement requirement. Both face the challenge of deploying AI in environments with significant prior investment in existing technology infrastructure.

The key difference is that healthcare AI governance has a direct patient-safety dimension that financial services AI governance does not. A biased credit model produces unfair financial outcomes. A biased clinical AI tool can produce incorrect diagnoses or inadequate care recommendations for certain patient populations. The stakes for AI failure are different, and the governance controls reflect that difference. FDA oversight introduces a device-regulation layer with no direct parallel in financial services. And the HIPAA framework creates a specific data protection baseline that, while narrower than full AI governance, is more mature and more consistently enforced than comparable data protection requirements in most financial services contexts.

For healthcare organizations that also have financial AI (patient financial assistance tools, revenue cycle AI, and insurance-facing AI), governance programs may need to bridge the two frameworks. The underlying methodology is similar: inventory, risk classification, accountability, monitoring, and documentation. The specific regulatory requirements differ by use case.

Where healthcare AI governance programs typically start

The most common starting point is not the most comprehensive governance framework. It is a structured assessment of the current state: which AI systems are deployed, their risk levels, what governance controls already exist, and where the gaps are relative to HIPAA, FDA, ONC, and EU AI Act requirements.

That assessment typically surfaces two categories of findings. The first is ungoverned AI that has been deployed without adequate controls, often with capabilities embedded in existing platforms and not subject to a formal procurement or clinical review process. The second is AI that has adequate technical controls but lacks organizational accountability, documentation, or monitoring.

The governance program that follows an assessment is usually structured in phases: immediate remediation for the highest-risk gaps; a governance foundation that establishes the inventory, accountability structures, and documentation standards; and ongoing monitoring and policy enforcement as the program matures.

Both Tristella's AI governance advisory and our healthcare IT strategy consulting practices contribute to this work. The AI governance expertise covers framework design, regulatory mapping, and policy infrastructure. The healthcare IT expertise covers the clinical environment: how AI tools integrate with EHR workflows, which clinical oversight processes need to be built or modified, and how to structure the governance program so that clinicians and health system leadership will actually use it.

The organizations that build the most durable healthcare AI governance programs are those that treat governance as a clinical and organizational capability, not a compliance exercise. The compliance deliverables are a byproduct of a well-run governance program. The governance program itself produces something more valuable: an organization that can deploy clinical AI with confidence, demonstrate accountability to regulators and enterprise customers, and identify problems in AI systems before those problems affect patients.

What to do next

If your organization is deploying clinical AI without a clear picture of your current governance posture, the right starting point is a structured assessment that maps your AI inventory to your regulatory exposure.

The Healthcare AI Adoption Scorecard provides a structured entry point for health systems and digital health companies that want to understand where they stand before designing a governance program. The AI Governance Gap Assessment goes deeper for organizations with a clearer picture of their AI portfolio and who want to assess the gap against specific regulatory frameworks.

For organizations working through a specific AI governance or healthcare IT challenge, a direct conversation is often the most efficient starting point.

Learn more about Tristella's AI governance advisory and healthcare IT strategy consulting, or contact us to discuss your organization's specific situation.

Related reading:

Sources: