US State Education Departments' automated essay scoring found biased against some groups
Automated essay scoring engines were used in many U.S. state standardized tests and multiple investigations and research studies found systematic differences in scores across demographic groups. Reporting and peer-reviewed analysis (including an ETS technical study) showed some engines gave higher average scores to certain groups and lower scores to others, and that some systems could be fooled by nonsense text.
Records by entity: US State Education Departments
State-run automated essay scoring used in high-stakes tests was reported and studied to produce demographic score disparities and to be susceptible to superficial gaming.
The models learned scoring shortcuts in the training data , measurable surface features that correlated with demographics rather than true writing quality.
Key facts
- What
- Automated essay scoring engines were used in many U.S.
- Incident date
- Jun 30, 2018
- Who
- US State Education Departments
- Failure mode
- Policy Violation
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
Beginning in at least 2018, many U.S. state education departments adopted automated essay scoring engines to grade written responses on standardized tests. Media investigations and technical reports documented that these systems produced mean score differences by demographic group and sometimes relied on surface features like sentence length and vocabulary rather than deeper meaning. Research published by ETS and reporting by outlets such as NPR and Motherboard/Vice described cases where the e-rater and other scoring engines gave systematically different scores for essays from different demographic groups and where some nonsense or formulaic essays could receive high automated scores.
What broke inside the model
The scoring systems were statistical models trained to predict human-assigned scores and therefore learned patterns present in training data that correlated with demographics rather than strictly with writing quality. The engines emphasized measurable surface features such as sentence length, vocabulary complexity, spelling, and grammar which can disadvantage English-language learners and other groups who write differently. In some deployments, a large share of essays were scored by the machine alone with limited human review, creating a pathway for biased or gamed scores to affect outcomes.
What it cost
Sources
- PressFlawed Algorithms Are Grading Millions of Students’ Essaysvice.com
- PressMore States Opting To 'Robo-Grade' Student Essays By Computernpr.org
- PrimaryUnderstanding Mean Score Differences Between the e-rater® Automated Scoring Engine and Humans for Demographically Based Groups in the GRE® General Test (ETS research report)onlinelibrary.wiley.com
Cite this entry
https://failureindex.ai/failures/state-education-departments-automated-essay-scoringAI Failure Index. "US State Education Departments' automated essay scoring found biased against some groups" (FI-0347). Realm Labs. https://failureindex.ai/failures/state-education-departments-automated-essay-scoring (indexed Jun 9, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0347. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.