Google flags parent's medical photo of his toddler as suspected child abuse
In February 2021 a San Francisco father took photos of his toddler’s swollen genital area for a doctor; those images were backed up to Google Photos and were later flagged by Google’s automated child sexual abuse material (CSAM) detection system. Google locked the user’s accounts and reported the matter to the National Center for Missing and Exploited Children, prompting a police inquiry that investigators later closed with no charges. The episode was reported publicly by The New York Times on 2022-08-21 and covered by other outlets.
An automated CSAM-detection pipeline misclassified a medical photo and triggered reporting and account removal.
Key facts
- What
- In February 2021 a San Francisco father took photos of his toddler’s swollen genital area for a doctor; those images were backed up to Google Photos and were later flagged by Google’s automated child sexual abuse material (CSAM) detection system.
- Incident date
- Feb 1, 2021
- Who
- Failure mode
- Agentic Action Error
- AI surface
- Computer Vision
- Severity
- High
What happened
In February 2021 a father (identified in reporting as "Mark") photographed his toddler’s groin at a nurse’s request and the images were backed up to Google Photos. Google's automated CSAM-detection tooling flagged the images, the company locked the user's Google accounts and reported the material to the NCMEC CyberTipline, and law enforcement opened an inquiry. Investigators later concluded the incident did not meet the elements of a crime and no charges were filed; the story was reported by The New York Times on August 21, 2022 and picked up by other outlets.
What broke inside the model
- 01 · TriggerA father photographs his toddler's infection at a doctor's request; the photos sync to Google.
- 02 · Model stepThe CSAM classifier flags the medical images as abuse imagery.
- 03 · Control gapEnforcement is fully automated: account termination and a report to NCMEC fire with no human review of context.
- 04 · FailureThe account, a decade of data, and the phone number are gone; police open an investigation.
- 05 · ConsequencePolice clear the father; Google still refuses reinstatement, proving the appeal path is part of the failure.
Google’s Content Safety tooling combines hash-matching of known CSAM with AI classifiers to identify potential new abusive imagery; in this case medical photos of a child were misclassified as CSAM. The automated pipeline resulted in account enforcement and an external report to NCMEC before the investigation was closed, showing a failure in the end-to-end decision and remediation process for flagged content.
What it cost
Sources
- PressA Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.nytimes.com
- PressGoogle AI flagged parents’ accounts for potential abuse over nude photos of their sick kidstheverge.com
- PressNYT: Parents Lose Google Accounts Over Abuse Image False Positivespcmag.com
- PressIncident 303: Google's Automated Child Abuse Detection Wrongfully Flagged a Parent's Photoincidentdatabase.ai
Cite this entry
https://failureindex.ai/failures/google-flags-parent-medical-photo-hisAI Failure Index. "Google flags parent's medical photo of his toddler as suspected child abuse" (FI-0407). Realm Labs. https://failureindex.ai/failures/google-flags-parent-medical-photo-his (indexed Jun 10, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0407. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
- AgentRealm
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.