At the HealthStack Conference 2026, Tiffani Bright, co-director of the Center for AI Research and Education at Cedars-Sinai Medical Center, warned that health care organizations must look beyond overall accuracy when evaluating AI systems.
Bright, who’s also an assistant professor of computational biomedicine, is recognized as the first Black woman in the U.S. to earn a doctorate in biomedical informatics.
During her session, “From Bias Evaluation to Trustworthy Healthcare AI Implementation,” she examined how gaps in validation and oversight can reinforce existing health disparities as AI adoption accelerates.
What Overall Accuracy Can Hide
An AI model might achieve 85% accuracy overall, but perform at only 70% accuracy for women, Bright explained. She illustrated how aggregate performance can conceal disparities affecting specific patient groups.
“You stop looking at the average,” she said.
More than 1,000 AI-enabled medical devices have received authorization from the Food and Drug Administration. Approximately 76% of those devices operate in radiology, according to research presented during the session.
“We’re not talking about it’s coming. We’re talking about it’s here,” Bright said.
Yet a 2024 study examining 692 FDA-authorized AI devices found that only 3.6% reported patients’ race or ethnicity in validation studies. Another 99% provided no socioeconomic information, while 82% did not report patients’ ages.
“Who was the tool tested on?” Bright asked.
Without that information, clinicians may struggle to determine whether AI systems were adequately evaluated among patients resembling those they treat.
“You can’t audit what was never reported,” she said.
When Algorithms Reinforce Disparities
Bright identified three points where bias can enter AI systems: data collection, model design and deployment. She recalled her experience conducting scientific evaluations at IBM Watson Health, where the technology’s promise raised difficult questions about whether its underlying data represented the patients it was intended to serve.
For Bright, the experience showed how powerful computing technology could still produce unreliable results when patient populations were inadequately represented. She cited a 2019 Science study involving an algorithm that used previous medical spending to determine which patients needed additional care.
Because Black patients historically incurred lower health care costs than white patients with comparable conditions, the algorithm underestimated their medical needs. Black patients consequently had to be sicker to qualify for the same additional services.
Bright also discussed research involving predictive models for perinatal mood and anxiety disorders among 19,430 patients. Researchers found that a fairness adjustment reduced disparities in screening predictions, but increased the model’s false-negative rate among minority patients. This illustrated how improving one measure of fairness can compromise another.
The findings reflected differences in model performance, not confirmed missed diagnoses in clinical practice.
For clinicians, the trade-off underscores the importance of evaluating multiple performance measures before relying on predictive tools to guide screening decisions.
Why AI Oversight Must Continue
The need for oversight extends beyond initial device authorization.
A 2024 review found that only 9% of examined AI devices were associated with prospective post-market surveillance studies.
“So that model that looks good today. Well, what’s gonna happen in six months?” Bright asked.
Changes in patient populations and clinical data can affect model performance over time, making continued evaluation necessary. Bright outlined three responsibilities for trustworthy AI governance: document how systems are evaluated, monitor performance across patient groups and assign an accountable individual.
“Somebody’s got to own it,” she said.
What Clinicians Can Do
Clinicians can begin by asking whether AI tools were validated among populations representative of their patients. They also can ask whether their institutions conducted local testing before deployment and how performance is monitored across demographic groups.
Research presented by Bright found that trained pathology professionals changed initially correct assessments to incorrect ones about 7% of the time after receiving erroneous AI recommendations. The finding illustrates automation bias, raising concerns that clinicians may abandon correct assessments when presented with inaccurate AI-generated advice.
Health professionals should document questionable AI outputs, report recurring problems through patient safety channels and challenge recommendations that conflict with clinical findings.
One attendee urged clinicians to establish AI governance committees and collaborate with community leaders rather than waiting until problems arise. Another questioned whether algorithmic bias reflects longstanding inequities in medical research, particularly the underrepresentation of people of color in clinical studies.
The discussion returned to the demographic reporting gaps identified earlier in the session. Improving representation, Bright explained, requires reconsidering how researchers recruit participants and engage communities. This includes ensuring people historically excluded from research have meaningful opportunities to participate.
For clinicians, the takeaway is to approach AI recommendations with the same scrutiny applied to other clinical evidence. That means questioning outputs that conflict with patient assessments, reporting potential errors and asking whether the technology has been evaluated among the populations they serve. As Bright’s presentation underscored, protecting patients requires more than trusting an algorithm’s accuracy; it requires knowing when to challenge it.
