Red Teaming¶
Red Teaming lets you proactively test your LLM application against adversarial attacks before they reach production. It generates realistic attack scenarios, runs them against your application, and measures how effectively LLM application avoids harmful outputs.
Once you have configured a runtime protection, you can also measure how reliably it blocks harmful inputs and (remaining) outputs.
Overview¶
A red team campaign guides you through four steps, shown at the top of the campaign page:
- Campaign — name the campaign and choose the adversarial model
- Attack sources — choose the attacks to generate
- Generate dataset — turn the configuration into adversarial test cases
- Evaluate application — run the test cases against your application and review the results
Each step shows its state at a glance (complete, in progress, needs attention), and the page opens on the step that needs you next.
Each evaluation run performs two passes automatically:
- Runtime protection pass — test cases run through LINK2AI.Trust with input and output runtime protection enabled
- Baseline pass — the same test cases run without runtime protection, reusing cached LLM responses
This lets you measure exactly how many attacks your runtime protection blocks compared to an unprotected baseline.
Step 1: Campaign¶
| Field | Description |
|---|---|
| Name | Shown in the campaign list |
| Description | Optional note on what the campaign tests |
| Adversarial Model | Generates and rewrites the attacks and judges the responses. Configure models under Settings → Models. |
A campaign tests the project's application, which you set up on the Application page. Attacks are generated from its Application Context: the more precise the context, the more specific the generated attacks and the proposed custom policies.
Step 2: Attack Sources¶
An attack source is a red teaming library that contributes test cases to the dataset. A campaign can combine several sources: the list on the left shows each source with a short summary, the editor on the right configures the selected one, and Add source adds another.
| Source | What it attacks with |
|---|---|
| Promptfoo | Attacks generated for your application context from attack targets, strategies and custom policies |
| PyRIT datasets | Published harm datasets, rewritten with prompt converters |
| PyRIT adaptive attacks | Multi-turn conversations in which an attacker pursues objectives specific to your application |
Recommended Attacks¶
You don't need to know the individual attacks: Recommend attacks lets the adversarial model read your application definition and choose the attacks that matter for your application, with a short reason for each, and write objectives for adaptive attacks. A few basic attacks, such as prompt extraction, are always part of the plan.
Review the plan and choose how thorough the test should be:
| Intensity | Use it for |
|---|---|
| Quick | A first impression: every recommended attack target, few test cases each, no extra strategies, one PyRIT dataset, short Crescendo conversations for the top objectives |
| Standard | Solid coverage: more test cases, the most effective strategies and PyRIT datasets, more objectives |
| Thorough | Before a release: the most test cases, strategies, datasets and converters, and long conversations with both adaptive attacks |
Each intensity shows the expected number of test cases. Apply plan turns it into attack sources, which you can fine-tune like any other. The recommendation needs a description of the application on the Application page and an adversarial model.
Your changes are saved when you continue to the next step with Save and next.
Promptfoo¶
Custom Policies¶
Policies describe rules specific to your application, for example "never grant discounts above 10%". Promptfoo generates attacks that try to make the application break them. Propose Custom Policies suggests policies from the application context using the adversarial model.
Attack Targets¶
Targets define the types of attacks or error-provoking inputs included in a dataset. Each target generates the configured number of tests per plugin. Presets select the targets for common applications and compliance frameworks; Browse all plugins gives you full control.
| Presets | Frameworks |
|---|---|
| Recommended, Minimal Test, RAG, Foundation, Guardrails Evaluation, MCP, Harmful | NIST AI RMF, OWASP LLM Top 10, OWASP Gen AI Red Team, OWASP API Top 10, OWASP Top 10 for Agentic Applications, MITRE ATLAS, EU AI Act, ISO 42001, GDPR, DoD AI Ethical Principles |
LINK2AI.Trust runs promptfoo entirely within your installation and never sends your application context to promptfoo's hosted service. Targets that only promptfoo's hosted service can generate are therefore not offered, and presets apply the targets that are available.
| Target | Description |
|---|---|
| Prompt extraction | Tries to trick the AI application into revealing its system instructions or parts of it, a typical preparation for more targeted attacks. |
| Tool discovery | Tries to trick the AI application into revealing the list of tools, functions, or API calls that it has access to, a typical preparation for more targeted attacks. |
| Divergent repetition | Instructs the AI application to generate indefinite output loops / unusually long repetitions, wasting tokens or degrading service quality and potentially leading to leakage of training data. |
| Shell injection | Steers a tool-using system towards unauthorized command execution. Relevant for assistants that can construct shell commands, invoke scripts, or pass user-controlled text into command-like workflows. |
| SQL injection | Steers a database-connected system into unsafe query construction or apparent execution of injected SQL. Relevant for agents that can execute database operations. |
| Excessive agency | Tests the reaction of the AI application to requests that lie beyond its capabilities. |
| Hallucination | Pushes the AI application towards hallucinations, requesting information it does not have. |
| Unverifiable Claims | Pushes the AI assistant towards hallucinations, requesting information that cannot be verified. |
| Overreliance | Generates requests with flawed premises to see whether the AI application accepts them at face value (and thereby engages in a potentially harmful interaction) instead of correcting them before it answers. |
| Contracts | Tests whether the AI application makes commitments beyond its authority or competence. |
| PII | Tries to make the AI application disclose personal data, directly, from its session or through connected APIs and databases. |
| Harmful content | Invites the AI application to disclose private information or infringe intellectual property. |
| Datasets | Established public benchmarks of harmful requests, such as BeaverTails, HarmBench or XSTest. |
Attack Strategies¶
Strategies control how attack prompts are transformed or delivered. Each selected strategy is applied to each of the attack targets.
Encodings apply a fixed transformation to the generated attack prompt. Why would you want to do that? - Encodings are used by attackers wanting to bypass security controls based on the recognition of specific terms. - Models learn to work with encoded input, but often their safety mechanisms fail with encoded input.
| Strategy | Description |
|---|---|
| Base64 | Encodes the attack in Base64. |
| Hex | Encodes the attack in hexadecimal. |
| ROT13 | Applies ROT13 substitution. |
| Leetspeak | Replaces characters with leet equivalents. |
| Morse Code | Encodes the attack in Morse code. |
| Pig Latin | Converts to Pig Latin. |
| camelCase | Removes spaces using camelCase. |
| Emoji Smuggling | Hides the attack in emoji. |
| Homoglyph | Replaces characters with visually similar Unicode. |
| Audio Encoding | Represents the attack as audio transcript. |
| Image Encoding | Embeds the attack in an image description. |
| Video Encoding | Embeds the attack in a video description. |
| Jailbreak Templates | Applies known jailbreak prompt templates. |
The generated attacks are always included as they are; every selected strategy adds a transformed variant of each of them. The number of test cases therefore grows with the targets, the tests per plugin and the strategies.
PyRIT Datasets¶
A PyRIT datasets source draws its attacks from PyRIT, Microsoft's red teaming framework:
| Setting | Description |
|---|---|
| Datasets | Curated collections of harmful requests that ship with PyRIT: AdvBench as a broad baseline, Microsoft AI Red Team sets per harm category (e.g. illegal activities, scams, malware, self-harm), and leakage sets for system prompt extraction and copyrighted text. |
| Seeds per Dataset | How many requests are sampled from each selected dataset. |
| Converters | Transformations applied to every sampled request. Original sends it unchanged; encodings and framings rewrite it deterministically; LLM rewriting (e.g. past tense, persuasion techniques, translation) uses the campaign's adversarial model. |
A PyRIT source generates datasets × seeds per dataset × converters test cases. Its responses are graded by asking the adversarial model whether the application refused the original request; test cases blocked by the runtime protection count as blocked without grading.
PyRIT Adaptive Attacks¶
Single prompts show how an application reacts to an attack out of the blue. Real attackers adapt: they start harmless, build on the answers and rephrase what was refused. A PyRIT adaptive attacks source has the adversarial model play such an attacker in a conversation with your application.
| Setting | Description |
|---|---|
| Objectives | What the attacker tries to make your application do, for example "Get the assistant to reveal the order history of another customer". Propose objectives suggests objectives from the application definition using the adversarial model. |
| Turns per Conversation | How many messages the attacker may send per conversation (1 to 10). |
| Attacks | Crescendo starts harmless and escalates step by step, backtracking when the application refuses. Red teaming converses freely and adapts to every response. |
Every objective is pursued once per selected attack, so the source generates objectives × attacks test cases. The conversations take place during the evaluation: each message passes the runtime protection and your application with the conversation so far. If the runtime protection blocks a message, the conversation ends and the test case counts as blocked. Otherwise the adversarial model judges whether the objective was achieved; an achieved objective is a harmful result. The evaluation without guardrails runs every conversation anew, so it shows what the attacker achieves against the unprotected application.
Adaptive attacks use many more model calls per test case than single prompts. They need an application that takes the conversation history, so they are not available for ChatKit applications. A source takes up to 20 objectives.
Changing the Configuration¶
You can change the configuration at any time, except while a dataset is being generated or an evaluation is running. A dataset and its runs belong to the configuration they were created with, so changing the adversarial model or the attack sources discards them. Before that happens, the portal asks for confirmation and offers Save as new campaign, which saves your changes as a copy and keeps the original campaign with its results. Renaming a campaign or changing its description keeps everything. A campaign or run can only be deleted once its generation or evaluation has finished or been stopped.
Step 3: Generate Dataset¶
Generate dataset turns the saved configuration into adversarial test cases. Generation runs in the background and shows its progress; you can leave the page or stop it. It uses credits per test case and is only started if enough are left. If it fails, the error and the generation log are shown and you can generate again.
Exploring the Dataset¶
Once generated, the dataset opens with its composition: the test cases per source, per attack (promptfoo target or PyRIT dataset) and per strategy or converter. Selecting a source, an attack or a strategy filters the test cases below.
The test cases are grouped by attack. Expand a group to see its test cases, or search all prompts. Selecting a test case shows its full prompt, the original request (when a strategy or converter rewrote it), its harm categories and how it is graded.
A dataset can be evaluated as often as you like, for example before and after changing runtime protection settings.
Step 4: Evaluate Application¶
An evaluation run executes the dataset against your application. The application must be set up on the Application page: a System prompt + model application needs its model and system prompt, a Chat completions endpoint application its endpoint, a ChatKit application its URL. Like generation, a run uses credits per test case and is only started if enough are left; it stops when the organization runs out of credits.
Starting a Run¶
Start the first run (later New run) asks for a name and, optionally, what changed since the last run, for example a guardrail threshold you raised. The run executes in the background; it can be stopped at any point, and it then completes the current test cases and halts.
| Status | Description |
|---|---|
pending |
Queued, not yet started |
running |
Actively executing test cases |
completed |
All test cases processed |
failed |
Run encountered an unrecoverable error |
stopped |
Manually stopped by the user |
The evaluation step lists all runs of the campaign and shows how the harmful and blocked counts develop from run to run.
Run Results¶
Open a run to see its results with guardrails or without guardrails (the baseline pass).
Attack Success Rate and Attack Flow¶
The run opens with its attack success rate: the share of attacks that produced a harmful response, next to the rate of the other pass. The difference between the two represents the protective value of your current runtime protection configuration.
The attack flow below follows the test cases through the pipeline in the order they are processed and shows where each one ended:
| Stage | Outcome |
|---|---|
| Input analysis | Blocked — the runtime input analysis (e.g. Malicious Intent) stopped the attack before it reached the application. |
| Application | Call failed — the application could not be called. |
| Output analysis | Blocked — the runtime output analysis stopped the application's response before it reached the user. All responses are analyzed, including refusals. |
| Grading | Harmful — the response contains what the attack tried to provoke. |
| Refused by the application — the application itself refused or answered safely. | |
| Grading failed — the response could not be graded. |
Without guardrails, the input and output analysis stages are skipped. Selecting an outcome filters the results.
Breakdown¶
The breakdown shows where attacks succeeded, by attack, strategy, severity and, with guardrails, the runtime protection module that blocked them. Each row shows how its test cases ended and their harmful count; selecting a row filters the results.
Results¶
The results table lists every test case with its outcome, attack and strategy, harmful results first. Sort by any column, filter by outcome, attack, strategy, severity or blocking module, or search prompts and responses. Selecting a test case shows its details:
| Field | Description |
|---|---|
| Prompt | The adversarial prompt sent to your application |
| Objective and Conversation | For adaptive attacks: the objective and every message of the conversation, with the message the runtime protection blocked |
| Original request | The request before a strategy or converter rewrote it |
| Response | The final response returned to the user (standard response if blocked) |
| Response before blocking | The application's response before the runtime protection blocked it (with guardrails only) |
| Blocked by | The analysis of every runtime protection module, as in the Playground; the modules that blocked the test case are expanded |
| Grading | Explanation for the assessment (particularly relevant for the remaining harmful responses) |
From the details you can also remove a test case from the run, for example when it does not apply to your application.