Health Data and AI: Biased Data, Biased Outcome

Can we trust the model? As a health analytics professional for decades, this is a question we ask ourselves daily. If we blindly trust an algorithm, we could have considered many patients “healthy” when they are not. There is a regulatory gap in managing data, not just devices or models. Current regulatory policies follow traditional statistical analysis, which is model-driven, but artificial intelligence (AI, or data science) models are data-driven. For example, value-based payment is “model dependent” [1]. Let’s start with how we can better govern data. Even though the One Big Beautiful Bill Act does not have a moratorium on states passing their laws regulating AI, it is still too early to tell whether there will be any future AI regulation enforcement pause or preemptive restrictions. The latest U.S. AI policy in “America’s AI Action Plan” focuses on “light regulation” and infrastructure. The latter would be a positive factor for AI development.

Mind the Regulatory Gap

Effective governance is critical to the safe and ethical use of data. The primary regulator for medical-related AI is the U.S. Food and Drug Administration (FDA), for whom clinical safety is the primary focus. The FDA oversees AI in the drug development process, Software as a Medical Device (SaMD) and AI-enabled devices. The second area of regulation comes from the Department of Health & Human Services (HHS), which is more focused on patients and health, as compared with the FDA, which is more clinically focused.

Some states also have medical regulations – most related to patient protection. For example, California has three healthcare-specific laws: AB-3030, SB-1223 and SB-1120. AB-3030 requires health providers to disclose the use of generative AI in patient communication; SB-1223 extends privacy law to include neural and biometric data as sensitive personal information; and SB-1120 requires AI or algorithm-based utilization management decisions to have a human clinical evaluation.

AI is a rapidly evolving field, and regulations need to balance safety and innovation. To paraphrase a famous line: the regulators are always planning for the last era. To properly regulate health AI, we need a new way of thinking about what’s best for patients and society. Furthermore, it is unclear how the liabilities are allocated among the AI creators, software and product developers, and users. An updated regulatory framework that aligns all stakeholders would be beneficial. My doctoral research is exploring the combination of responsible AI and equitable health policy.

Hidden Bias in Health Data

AI tools are only as good as their training data. Large language models (LLMs) such as ChatGPT have the potential to solve two key issues in AI health tech. First, language is context-dependent. Conversational AI (e.g., LLM) allows users to interact with the machine, ask clarifying questions and solve problems collaboratively. The second challenge is that users need to understand and trust the reasoning behind AI recommendations. Explainable AI will show and explain its logic and rationale.

Sisyphus of AI and the Rock of Healthcare

Health AI is the Mr. Godot in “Waiting for Godot”: We have been told that he will come very soon, but just not yet. Healthcare is a very complex field, and it is helpful to go back to the basics of knowledge acquisition. There are four ways to acquire knowledge:

1. We can ask the patients (e.g., a survey) and face nonresponse bias.
2. We can observe (e.g., tracking devices) and face nonobservable bias.
3. We can infer from third-party data (e.g., administrative claims data), but spurious correlations may be present.
4. We can use theoretical or empirical models and possibly encounter modeling bias.

There is simply no single best way.

Presumed Evidence

A more subtle issue in health data is hidden assumptions and preconceived notions. How can a model recognize unknown patterns (known as out-of-sample properties in statistics)? How will the machine diagnose a patient if it has no data for that specific group? It is common to presume group behaviors are the same for individuals (known as the ecological fallacy).

Confirmation bias, which is closely related to publication bias, states that we often find evidence to support our preconceived notions, and that is why we seldom see a failed use case in any new product. This limits the universe of training data for AI models.

One of the landmines of adopting health data is the potential misuse of data that is collected for a different purpose. Often, we use what we have, not what we need. As illustrated in the Optum algorithm racial bias controversy, using only available data could lead to unintended consequences [2].

For example, in an arrangement such as a value-based care contract, the providers may not have a strong incentive to fully document all details (more focus on outcome). Additional documentation could cost money. As a result, the data might be biased with unknown missing data. A standard disclosure of data capture and data use would help to increase health data transparency.

A New Social Contract with AI

Without better data governance, AI tools could bring great harm. One of the ironies of history is that we will see things only when it is over. That’s why we must be both forward and backward looking. To chart a voyage for Responsible AI, we should recognize how centuries of human inquiry have shaped our understanding of reason, ethics and methodology. A comprehensive regulatory framework that establishes “ethical use” standards for health data and a voluntary certification or reporting requirement for responsible AI use would be helpful. I am also trained in evidence-based healthcare, so I would suggest a public-private policy advisory board similar to the NICE (National Institute for Health and Care Excellence) in the U.K., which could be a promising start toward an ethical and equitable AI world.

Concrete Policy Actions

To summarize, here are three areas in which better AI regulation could help:

1. FDA – Expedite SaMD evaluation with a focus on data-use disclosure in AI-enabled models based on risk assessment. This will be particularly helpful to tech startups.
2. States – Harmonize state-level requirements on AI usage in the U.S., such as clinical decision support or utilization management, to improve development efficiency, establish bias audit standards and mandate disclosure.
3. Congress – Develop, clarify and simplify AI safety and safe harbor policies on “extended” use (i.e., not formally assessed) and model liability attributions.

References

1. R. Wells, W. Walters and R. Hughes IV, 2025, “CMS’ pay-for-performance paradox,” Health Affairs, June 16, http://healthaffairs.org/content/forefront/cms-pay-performance-paradox.
2. Gawronski, 2019, “Racial bias found in widely used health care algorithm,” NBC News, November 7, https://www.nbcnews.com/news/nbcblk/racial-bias-found-widely-used-health-care-algorithm-n1076436.