GPT-4o Chinese token library polluted by spam and pornography
OpenAI's GPT-4o model was found to have a Chinese token library polluted with spam and pornographic phrases. This resulted from inadequate data cleaning of the training corpus, allowing glitch tokens that could cause hallucinations or be used for jailbreaking.
The GPT-4o tokenizer absorbed spam and pornographic phrases due to inadequate data cleaning, creating glitch tokens that could be used to jailbreak the model.
Key facts
- What
- OpenAI's GPT-4o model was found to have a Chinese token library polluted with spam and pornographic phrases.
- Incident date
- May 13, 2024
- Who
- OpenAI
- Failure mode
- Brand & Safety Incident
- AI surface
- Chatbot
- Severity
- Medium
What happened
Shortly after the release of GPT-4o, researchers discovered that its Chinese token library contained an abundance of tokens consisting of gambling and pornographic phrases. These tokens caused the model to produce irrelevant responses or hallucinate. In some cases, they were used to bypass safety guardrails.
What broke inside the model
- 01 · TriggerA user prompts the model in public view.
- 02 · Model stepThe model produces unsafe or off-brand output.
- 03 · Control gapNo filter holds the line before publish.
- 04 · FailureThe output goes public unchecked.
- 05 · ConsequenceA reputational or safety incident lands.
A contained signal crosses into output that goes public.
The failure was caused by inadequate cleaning and filtering of the training corpus used for the o200k_base tokenizer. This allowed phrases from content hijacking spam websites to be encoded as valid tokens. These polluted tokens acted as glitch tokens that triggered undefined behaviors in the model.
What it cost
Sources
- PressGPT-4o’s Chinese token-training data is polluted by spam and porn websitestechnologyreview.com
- PrimaryGlitch Tokens in GPT-4o: Seeking Clarificationcommunity.openai.com
Cite this entry
https://failureindex.ai/failures/gpt-chinese-token-library-polluted-spamAI Failure Index. "GPT-4o Chinese token library polluted by spam and pornography" (FI-0647). Realm Labs. https://failureindex.ai/failures/gpt-chinese-token-library-polluted-spam (indexed Jun 22, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0647. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.