Data poisoning is an attack on the data a machine learning model learns from. The attacker slips crafted samples into training, fine-tuning or retrieval data so the model learns a behavior they chose, often one that only shows up when a trigger appears. If your team fine-tunes models, runs retrieval over internal documents or pulls weights from a public hub, this is a supply chain problem for you, and the uncomfortable part is how little poisoned data it takes.
Start with the number that should change how security teams think about this. In October 2025, researchers from Anthropic, the UK AI Security Institute and The Alan Turing Institute showed that 250 malicious documents were enough to backdoor language models from 600M to 13B parameters. The largest of those models trained on more than 20 times as much clean data as the smallest, and the attack worked about as well on both. Two hundred and fifty documents is an afternoon of work. We will come back to that number, because almost every defensive decision in this post depends on whether you believe it applies to you.
What is a data poisoning attack?
A data poisoning attack corrupts a model’s behavior by corrupting what it learns from, rather than by attacking the deployed software. OWASP lists it as LLM04:2025, Data and Model Poisoning, and frames it as an integrity attack that can hit pre-training, fine-tuning or embedding data.
Two goals cover most cases. The blunt one degrades or skews output in general, which tends to get noticed. The dangerous one is a backdoor, where the model behaves normally until a trigger phrase or condition appears and then does what the attacker trained it to do. In the 250-document study the trigger was a short tag, the word SUDO in angle brackets, and the backdoored behavior was gibberish output, deliberately low stakes so it could be measured cleanly. Anthropic’s earlier Sleeper Agents work used a nastier example, a model that writes secure code when the prompt says the year is 2023 and inserts exploitable code when it says 2024.
That difference matters for defenders, because a backdoor is built to pass every test that does not contain its trigger.
Why a few poisoned samples are enough
The old mental model said an attacker had to control some percentage of the training set, which made big models feel safe by sheer volume. The October 2025 paper argues the opposite, finding that poisoning needs a near-constant number of documents regardless of dataset size. In their runs, 100 poisoned documents were not enough to reliably backdoor any model, while 250 or more reliably succeeded across every size they trained. Those 250 documents were roughly 420k tokens, about 0.00016% of the training data.
Medicine supplies a sharper version. A team publishing in Nature Medicine on 8 January 2025 replaced tiny fractions of The Pile with generated medical misinformation hidden in web pages and trained models on the result. In their 1.3B models, harmful completions rose by 11.2% at 0.01% poisoned tokens and still by 7.2% at 0.001%. For a 4B model, swapping one million of 100 billion tokens for vaccine misinformation raised harmful content by 4.8%, using 2,000 fake articles that cost US$5.00 to generate.
The finding that worries us most from that study is a quieter one. The poisoned models scored comparably to clean ones on all five medical benchmarks, so the standard evaluation would have shipped them.
Data poisoning in AI, with examples from the three places it gets in
Most organizations do not pretrain a model, which is why the 250-document result can feel distant. Look instead at where your team takes in data it did not write, because each of those is a surface. We group them into three.
Fine-tuning and training data
This is the classic surface, and it is larger than it looks because training sets are assembled from the web. Carlini and colleagues showed that an attacker could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for $60 because web content can change after a dataset’s index is published, so what the curators reviewed is not what later downloaders get. They call it split-view poisoning. A second technique, frontrunning, edits crowd-sourced pages such as Wikipedia just before a scheduled snapshot. The Anthropic study also reports the same near-constant dynamics during fine-tuning, which is the stage most enterprises actually run. In June 2026, researchers added a twist for multi-stage pipelines, showing that attackers who each poison a different post-training stage can succeed together when neither looks dangerous alone.
Retrieval corpora (RAG)
If your assistant answers from a knowledge base, whoever can write to that knowledge base can steer the answers. PoisonedRAG, presented at USENIX Security 2025, reached a 90% attack success rate by injecting five malicious texts per target question into a database holding millions of texts, and the defenses the authors evaluated were insufficient. No training run is needed, so the poison works as soon as the document can be retrieved. In our experience this is the surface teams overlook, because the wiki, ticketing system or shared drive feeding retrieval was never classified as model input.
Downloaded models
The third surface is the model file itself. In July 2023, Mithril Security edited GPT-J-6B to state a false fact and uploaded it to Hugging Face under a lookalike organization name, and the edited model differed from the original by only 0.1% accuracy on the ToxiGen benchmark. In February 2024, JFrog found around 100 models on the same hub whose files ran code on load through Python pickle, one of them opening a shell for the attacker. Strictly, the second case is malware rather than poisoning, but OWASP files both under the same risk, because the defender’s question is identical. Do you know what is inside the weights you just loaded?
How to detect data poisoning
Detection is hard because a good backdoor hides behind its trigger, and the research is candid about it. Sleeper Agents found that safety training did not remove the backdoor and that adversarial training could teach the model to recognize its trigger better, creating a false impression of safety. The July 2026 ToxScreen benchmark of roughly 800 backdoored models concluded that no method reliably surfaces every backdoor. It did find that ranking candidate tokens by attack success recovered the trigger wherever the backdoor was effective, and that a model which is broadly easy to jailbreak is itself a warning sign.
So our position is to detect at the data layer first and treat model-level testing as a second net. In practice that means watching for these signals:
- A training or fine-tuning dataset that changed outside change control, or whose hash no longer matches the version that was reviewed.
- Retrieval sources receiving new or edited documents from accounts that do not normally write there, especially documents that read like answers to specific questions.
- A model artifact pulled from an unpinned revision, a lookalike publisher name, or a pickle-based format.
- Evaluation results where outputs shift sharply when an unusual token or phrase is added to otherwise clean prompts, which is how the 250-document study measured its backdoor.
- Domain answers that contradict a trusted reference. The Nature Medicine team’s knowledge-graph cross-check caught over 90% of the misinformation in text from their poisoned models.
None of these requires reading a model’s weights, which is the point. They sit in logs and pipelines you already own.
Mitigation: data poisoning attack prevention
Prevention follows the same three surfaces. For training and fine-tuning data, OWASP recommends tracking data origins and transformations with CycloneDX or an ML-BOM, vetting data vendors and versioning datasets with tools like DVC so a reviewed snapshot is the one that trains. Carlini’s split-view attack works because the dataset index and the downloaded content drift apart, which is why their paper recommended several low-overhead defenses to the dataset maintainers they notified. On your side the equivalent is hashing what you reviewed and refusing anything that does not match.
For retrieval, treat write access to every indexed source as access to the model’s answers, and review it with the same care. For downloaded models, pin exact revisions, prefer the safetensors format that Hugging Face built as a safer alternative to pickle, and scan before load. Then test for triggers as part of red teaming; our guide to AI red teaming covers how to scope that work, and our prompt injection guide covers the inference-time attack that poisoned retrieval content often delivers. For the governance side, the joint CISA, NSA and FBI guidance on AI data security from May 22, 2025 covers data security and integrity risks across every phase of the AI lifecycle, and NIST AI 100-2 E2025 gives your team shared terminology for the attacks.
If you are building retrieval or fine-tuning pipelines and want someone to map these surfaces with you, that is the work our secure AI deployment practice does.
Frequently asked questions
What is data poisoning in simple terms?
It is tampering with the data an AI model learns from or retrieves, so the model behaves the way an attacker wants. The change can be broad, such as making answers less accurate, or narrow, such as a hidden backdoor that activates only when a specific trigger phrase appears in the prompt. OWASP tracks it as LLM04 in its 2025 Top 10 for LLM applications.
What is the difference between data poisoning and prompt injection?
Timing and persistence. Prompt injection arrives at inference time inside the input and affects that interaction. Data poisoning happens earlier, in training, fine-tuning or the retrieval corpus, and it persists until the data or model is replaced. The two meet in retrieval systems, where a poisoned document can carry an injected instruction to every user whose question retrieves it.
Can data poisoning be detected?
Partly. A July 2026 benchmark of roughly 800 backdoored models found no method that reliably surfaces every backdoor, so the strongest detection lives upstream, in dataset hashes, change control, write-access logs on retrieval sources and checks on model provenance. Trigger-search testing and cross-checking answers against trusted references catch some of what slips past those controls.
Does data poisoning affect companies that only use hosted models?
Less at the training layer, yes, but retrieval is still yours. If your assistant answers from your documents, anyone who can edit those documents can shape its answers, and PoisonedRAG reached 90% success with five texts per question. Any fine-tuning you do on top of a hosted model also brings the training surface back into your scope.
Bottom line
Back to the 250 documents. Even if the exact number does not hold for frontier models, the lesson does: poisoning scales with the attacker’s access to your data, not with the size of your model. List every source that feeds training, fine-tuning and retrieval this week, name an owner for each, and pin and hash what you load.