AI red teaming is adversarial testing of an AI system, the model plus everything wired around it, to find the ways it can be pushed into causing real harm before an attacker or an unlucky user finds them first. Microsoft’s AI Red Team, which formed in 2018 and has since red teamed more than 100 generative AI products, describes the job in one sentence: they break the technology so others can build it back stronger. If you are shipping an LLM application or an agent that reads content outsiders can write, you need this practice, and you need it to keep running after launch.
Picture the case Microsoft uses to explain its method. A copilot summarizes a user’s inbox. A scammer sends an email with a hidden instruction telling the copilot to ignore its previous instructions and present a malicious link. In Microsoft’s paper on those 100 products, that one email is a full test case: an actor, a technique, a weakness and an impact. We will return to that inbox, because most decisions in an engagement show up in it.
What is AI red teaming, and how is it different from a benchmark?
The short answer is that a benchmark measures known harms on a fixed dataset, while a red team goes looking for the harms nobody has written down yet. Microsoft lists “AI red teaming is not safety benchmarking” as one of its eight lessons. Benchmarks make it easy to compare models on the same questions. They cannot tell you what happens when your particular model summarizes your particular inbox with your particular connectors attached.
The other difference is the target. Most adversarial machine learning research optimizes inputs against the model itself, yet Microsoft’s second lesson quotes the line that “real attackers don’t compute gradients, they prompt engineer”. In their operations, basic techniques often worked as well as gradient-based ones, and sometimes better, because the model is one component of a larger system and the system is where the weak joints are. So the scope of an AI red team is the application, its tools, its data paths and its infrastructure, not just the weights.
How to scope an AI red team engagement
Start from impact, not from a list of jailbreaks. Microsoft’s first lesson says that starting from potential downstream impacts makes it more likely an operation produces findings tied to real risk, and then the team works backwards to the paths an adversary could take. Two questions drive that. What can the system do, and where is it applied? The same model as a creative writing assistant and as a summarizer of patient records carries very different risk, and the paper uses exactly that example.
Capability matters in a less obvious way too. Larger models understand encodings such as base64 and ASCII art, which means they can also follow instructions smuggled in those encodings, while a smaller model may not understand them at all. Microsoft calls the smaller model “capability constrained” and treats testing it for encoding attacks as wasted effort. We agree, and push it further for agents. Write down every tool the agent can call and every data source it reads, then rank them by the worst thing one hijacked call could do. That list is your scope.
For our inbox copilot, the scope document is short and concrete.
| Ontology element | Inbox copilot example | What the red team asks |
|---|---|---|
| System | Copilot with mailbox read access and link output | What else can it reach (calendar, files, send mail)? |
| Actor | External scammer; also a benign user who hits a failure | Who can put text in front of the model? |
| Technique | Cross-prompt injection in an inbound email | Which encodings and phrasings does this model follow? |
| Weakness | Model cannot separate instructions from email content | Which control is supposed to stop the action? |
| Impact | User clicks a malicious link; data leaves the tenant | What is the worst single action, and is it logged? |
The five columns come straight from Microsoft’s threat model ontology, which maps techniques to MITRE ATT&CK and MITRE ATLAS where it can. The ontology also covers benign actors, because many embarrassing outcomes come from ordinary prompts.
LLM red teaming versus red teaming AI agents
Scoping a chatbot and scoping an agent look similar on paper, and yet the results diverge sharply once the system can act. A chatbot that says something harmful is a content problem. An agent that reads a poisoned web page and then sends email, moves files or runs code is a security incident. NIST’s Center for AI Standards and Innovation (CAISI) calls this agent hijacking, a form of indirect prompt injection, and frames it as the old problem of a system that cannot separate trusted instructions from untrusted data.
The public evidence on agents is sobering. A competition run by Gray Swan and published in July 2025 put 22 frontier AI agents through 44 realistic deployment scenarios. Participants submitted 1.8 million prompt-injection attacks and more than 60,000 of them produced policy violations such as unauthorized data access and illicit financial actions. Nearly all agents violated policy for most behaviors within 10 to 100 queries, and resistance to attack showed limited correlation with model size, capability or inference-time compute. A later competition analysed with CAISI and the UK AI Security Institute found at least one successful attack against every one of the 13 frontier models tested, across more than 250,000 attempts from over 400 participants.
The lesson we take for our own engagements is that picking a stronger model does not close the gap. If our inbox copilot gained a “send email” tool, the scope document above would need a new impact row before anyone argued about which model to run. We covered the design side of that problem in our breakdown of agentic AI security risks.
Why one attempt per attack undercounts risk
This is where internal tests most often mislead, because the number looks precise. Language models are probabilistic, so a payload that fails once can succeed on the next try, and an attacker who pays nothing per attempt will simply retry. CAISI tested this directly. Taking five injection tasks against an agent built on the upgraded Claude 3.5 Sonnet, the average attack success rate was 57% on a single attempt and 80% when each attack was tried 25 times.
Static attack libraries undercount in a second way. The upgraded model resisted previously tested attacks far better than its predecessor, which looked like progress. Then CAISI and UK AISI red teamers built new attacks for that specific model, and attack success rose from 11% for the strongest baseline attack to 81% for the strongest new one.
| Period | Value |
|---|---|
| Strongest baseline attack | 11% |
| Strongest new red-team attack | 81% |
So report results the way CAISI recommends, per task and over repeated attempts, not as one aggregate pass rate. A data exfiltration task with a low success rate still deserves attention, because the impact of a single success varies widely between tasks. And budget for adaptive attacks written against your system, because an attacker will write them.
AI red teaming tools, and where people still matter
Automation is what makes repeated attempts affordable. Microsoft built PyRIT for this, and the paper describes prompt datasets, encoding converters, automated attack strategies like TAP, PAIR and Crescendo, and scorers. Together they cover far more of the risk surface than manual probing and help estimate how often a failure occurs. NVIDIA’s garak probes a model or dialog system in a structured way to surface weaknesses. Promptfoo generates adversarial inputs against an application and is often wired into CI/CD so the probes run on every change. These are examples to evaluate, not a ranking.
Tools generate and score. They do not decide what matters. Microsoft found that using an LLM to judge outputs works for simple categories but is less reliable in specialized domains such as medicine, cybersecurity and CBRN, where subject matter experts had to make the call. Their advice is to scale with automation and keep humans in the loop. In practice that means a person designs the system-level attack chains and reviews every high-impact finding, while the tooling handles volume and repetition.
The findings are often ordinary application security
Red teams that only test prompts miss half the report. Microsoft’s seventh lesson is that LLMs amplify existing security risks as well as adding new ones, and one of its published case studies is a server-side request forgery in a video-processing generative AI application. The paper lists the usual suspects alongside it, from outdated dependencies to credentials in source and missing input and output sanitization.
That is why the OWASP GenAI Red Teaming Guide, published January 22, 2025, splits the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. Our inbox copilot illustrates the point well. The injection is the headline, but the link only becomes harmful if output handling, egress controls and logging also fail. Each of those is a classic test your pentest team already runs.
Mitigation: turning AI red team findings into fixes
A finding is only useful if it changes the system, and here Microsoft’s working assumption is the one we would put on every engineering wall. If an LLM is supplied with untrusted input, assume it will produce arbitrary output. Fixes therefore belong outside the prompt wherever possible: limit which tools a hijacked agent could reach, require confirmation for irreversible actions, filter what leaves the system, and log every tool call so the next attempt is visible. Our prompt injection defense guide goes deeper on those controls.
Then rerun the same attacks, plus new ones, after each fix. Microsoft frames this as break-fix cycles that never fully end, with the goal of raising the cost of a successful attack rather than declaring the system safe. Regulators are moving the same way. For providers of general-purpose AI models with systemic risk, the EU AI Act requires conducting and documenting adversarial testing as part of model evaluation. When this testing needs outside hands, it is part of the LLM and agent security work we do.
Frequently asked questions
What is AI red teaming in simple terms?
It means assigning people to attack your AI system the way a real adversary or a careless user would, so you find harmful behavior before production does. Unlike a benchmark, it adapts to your application, tools and data, and it covers security impacts such as data theft as well as harmful content.
How is LLM red teaming different from a penetration test?
A penetration test focuses on infrastructure and application flaws. LLM red teaming adds attacks that work through language, such as prompt injection and jailbreaks, and judges outputs that are probabilistic. A good engagement does both, because Microsoft’s case studies show classic flaws like SSRF sitting next to the AI-specific ones.
Which AI red teaming tools should we start with?
Open-source frameworks such as PyRIT, garak and Promptfoo cover most first engagements. Pick the one that fits how you ship: a scripted framework for deep manual campaigns, a CI-friendly tool for regression testing on every change. None of them replaces a person deciding which impacts matter.
How often should we red team an AI system?
At minimum before launch and after every model swap, prompt change or new tool connection, since each can change behavior. CAISI’s findings show defenses that hold against previously tested attacks can fail against attacks written for the new system, so treat red teaming as a recurring control rather than a one-time audit.
Bottom line
Scope your first AI red team engagement from the worst action your system can take, test the whole application and not just the model, and report success rates per task over repeated attempts. Then fix outside the prompt, rerun the attacks, and keep the loop on the release calendar.