Google Health diabetic retinopathy AI fails in real world clinic settings
Google Health's AI for detecting diabetic retinopathy failed to maintain its laboratory accuracy when deployed in real world Indian clinics. The system was hindered by suboptimal environmental conditions and data quality issues.
The AI performed well in the lab but failed in the field due to a lack of robustness against real world image quality.
Key facts
- What
- Google Health's AI for detecting diabetic retinopathy failed to maintain its laboratory accuracy when deployed in real world Indian clinics.
- Incident date
- Dec 1, 2019
- Who
- Google Health
- Failure mode
- Agentic Action Error
- AI surface
- Computer Vision
- Severity
- High
What happened
Google Health deployed an AI system to screen for diabetic retinopathy in clinics in India. While the system performed well in controlled lab settings, it struggled with poor lighting, low quality images and unreliable internet in real world environments. This led to a high rate of false negatives and missed diagnoses in actual clinical practice.
What broke inside the model
- 01 · TriggerAn agent plans a multi-step task.
- 02 · Model stepIt chooses a wrong or destructive action.
- 03 · Control gapNo confirmation gate guards the write.
- 04 · FailureThe action commits to a system of record.
- 05 · ConsequenceData is changed or destroyed irreversibly.
A wrong action commits, and the step is written before anything can stop it.
The model suffered from a lack of robustness to real world data variance, specifically failing when confronted with image quality and lighting conditions not present in the training set. The system also lacked an effective fallback mechanism for poor quality images, often providing unreliable results instead of flagging the image as ungradable.
What it cost
Sources
- PressGoogle's medical AI was super accurate in a lab. Real life was a different story.technologyreview.com
- PressGoogle medical researchers humbled when AI screening tool falls short in real-life testingtechcrunch.com
Cite this entry
https://failureindex.ai/failures/google-health-diabetic-retinopathy-fails-realAI Failure Index. "Google Health diabetic retinopathy AI fails in real world clinic settings" (FI-0361). Realm Labs. https://failureindex.ai/failures/google-health-diabetic-retinopathy-fails-real (indexed Jun 9, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0361. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
- AgentRealm
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.