New York City Department of Education releases disputed value-added teacher ratings
The NYC DOE released "value-added" ratings that attempted to quantify teacher effectiveness using a statistical model. The ratings were widely criticized for being imprecise and unreliable, leading to disputes over their use in personnel decisions.
The ratings were described as an imprecise science.
Key facts
- What
- The NYC DOE released "value-added" ratings that attempted to quantify teacher effectiveness using a statistical model.
- Incident date
- Feb 25, 2012
- Who
- New York City Department of Education
- Failure mode
- Policy Violation
- AI surface
- Algorithmic Decision
- Severity
- Medium
What happened
The New York City Department of Education released "value-added" reports that purported to estimate teacher effectiveness through statistical analysis of student test scores. These ratings were distributed to news outlets and the public, identifying thousands of teachers' performance levels. The move sparked widespread backlash, as the estimates were viewed as imprecise and unfair metrics for professional evaluation.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The system used a Value-Added Model (VAM) to predict student gains based on demographics and previous performance. The failure was a lack of statistical reliability, where the model's margin of error was too large to distinguish actual teacher impact from random noise.
What it cost
Sources
- PressTeacher Quality Widely Diffused, NYC Ratings Indicatenytimes.com
- PressN.Y.C. Officials Release 'Value Added' Reportsedweek.org
Cite this entry
https://failureindex.ai/failures/york-city-department-education-releases-disputedAI Failure Index. "New York City Department of Education releases disputed value-added teacher ratings" (FI-0676). Realm Labs. https://failureindex.ai/failures/york-city-department-education-releases-disputed (indexed Jun 22, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0676. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
Realm compares what the model is about to output or do against the policy that governs the deployment, in real time, and can deny or redact the action before it takes effect, which is the gap an after-the-fact review never closes in time.