Researchers find systemic racial bias in PredPol crime forecasting software
A 2016 study revealed that PredPol's predictive policing software produced biased outputs that disproportionately targeted minority communities. The findings indicated that the AI reinforced existing policing patterns rather than predicting actual crime levels.
The algorithm created a runaway feedback loop where police presence generated the very data used to justify more police presence.
Key facts
- What
- A 2016 study revealed that PredPol's predictive policing software produced biased outputs that disproportionately targeted minority communities.
- Incident date
- Oct 7, 2016
- Who
- PredPol
- Failure mode
- Policy Violation
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
Research by Kristian Lum and William Isaac demonstrated that PredPol's algorithms led police to patrol already overpoliced communities more frequently. This created a cycle where the software predicted crime in areas where police were already making more arrests, regardless of actual crime rates. The bias was observed in simulations involving drug arrest data in Oakland, California.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The system suffered from a runaway feedback loop. Because the model relied on historical arrest data rather than actual crime occurrence, it predicted future crime in areas that had been heavily policed in the past, leading to more arrests in those same areas and further biasing the training data.
What it cost
Sources
- PrimaryTo predict and serve?rss.onlinelibrary.wiley.com
- PressPredictive policing algorithms are racist. They need to be dismantled.technologyreview.com
- Court FilingAlgorithms in Policing: An Investigative Packetlaw.yale.edu
Cite this entry
https://failureindex.ai/failures/predpol-predictive-policing-software-shows-systemicAI Failure Index. "Researchers find systemic racial bias in PredPol crime forecasting software" (FI-0331). Realm Labs. https://failureindex.ai/failures/predpol-predictive-policing-software-shows-systemic (indexed Jun 9, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0331. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.