LLM Red Teaming: What Gets Tested, What Gets Found, and What a Real Report Contains

System-Level Testing vs Content Safety Probing

There are two things sold under the name "AI red teaming" and they are not close relatives.

The first is content safety probing: someone spends a week trying to make a chatbot say something embarrassing, and delivers a deck of screenshots. It has a use — reputational risk is real — but it tests the model, and the model is not usually where your risk lives.

The second tests the system. It asks what an attacker can make your deployment do: which tools it can be induced to call, which data it can be persuaded to retrieve and disclose, which downstream systems inherit its privileges, and what happens when instructions arrive from a document rather than from a user. That is a security assessment. The findings are exploits with a blast radius, not quotes.

The distinction matters more in 2026 than it did a year ago, because the systems being deployed have changed. A chatbot that answers questions has a narrow failure surface. An agent that reads your ticketing system, writes to your CRM, and calls internal APIs on a schedule has the failure surface of a service account with a language model attached — one that follows instructions found in whatever it reads.

Settled Taxonomies to Test Against

For several years, AI security assessment lacked a shared vocabulary, which made scoping arguments unwinnable and reports incomparable. That has largely resolved.

• The OWASP Top 10 for LLM Applications: Covers prompt injection, sensitive information disclosure, improper output handling, supply chain exposure, excessive agency, and more. The 2026 edition maps its entries directly to NIST, MITRE ATLAS, and CWE.
• The OWASP Top 10 for Agentic Applications (ASI01 to ASI10): Published by the OWASP GenAI Security Project in December 2025, covering agent planning and goal manipulation, tool misuse, agent identity and impersonation, supply chain, unsafe code execution, memory/context poisoning, inter-agent communication, and rogue agent behaviour.
• NIST AI RMF (AI 600-1): Supplies the generative AI risk management profile.
• ISO/IEC 42001: Supplies the management system a certification body will formally assess.

What a System-Level Assessment Actually Probes

1. Indirect Prompt Injection

The highest-yield class of finding, consistently. The user is not the attacker. The attacker is whatever the system reads: a support ticket, a PDF in a shared drive, a webpage, a calendar invite, an email signature, or a code comment.

If your agent retrieves external content and that content reaches the model in the same channel as its instructions, an attacker who can influence any of that content can influence the agent's behaviour.

2. Tool and Permission Boundaries

Agents act through tools. Every tool is a permission grant, and permission grants compound. Which tools can this agent call? What identity does it call them with? Can the agent chain tools to reach a capability it was never explicitly granted — read one system, write to another, and trigger a third?

3. Memory and Context Poisoning

Agents that persist state across sessions can be taught things. Content written into memory during one interaction may influence behaviour in another, potentially for a different user. This is a stored-injection problem with an unusually long dwell time.

4. Output Handling Downstream

If model output is rendered in a browser, executed as code, passed to a shell, or inserted into a query, the model is now an untrusted input source feeding classic injection and XSS vulnerability classes.

5. Multi-Agent and Inter-Agent Trust

Where agents call other agents, or communicate over emerging agent protocols, the trust relationships between them are usually implicit and unvalidated. One compromised agent in a chain can propagate malicious instructions to the others.

Further reading: Microsoft 365 Copilot Security Review: What to Check Before You Switch It On and Shadow AI: How to Find, Assess and Govern Unsanctioned AI Use in Your Organisation.

Sources: OWASP GenAI Security Project, Top 10 for LLM Applications and Top 10 for Agentic Applications (ASI01–ASI10, published December 2025); NIST AI RMF and Generative AI Profile (AI 600-1); Regulation (EU) 2024/1689 and Regulation (EU) 2026/1744; Cyber Security and Resilience Bill, UK Parliament.