Model Evaluation for Extreme Risks: What Red-Teaming Tests
Red-teaming is often treated as a headline safety guarantee. Looked at closely, it is a narrow, adversarial sampling exercise, and confusing it with a risk bound is where most evaluation claims go wrong.
Why the framing matters
Whenever a lab announces that a model has been red-teamed for extreme risks, such as helping someone plan a biological, chemical or cyber attack, the phrase does a lot of quiet work. It sounds like a safety audit, something exhaustive and certifying. In practice, red-teaming is a search procedure. A group of people, sometimes domain experts, sometimes generalist testers, try to elicit harmful outputs from a model within a fixed time budget, using a fixed set of prompting strategies. What comes out is a list of successful and unsuccessful attempts. That list tells you something real, but it is much narrower than the word 'evaluation' usually implies.
The core confusion is between finding a failure and bounding the probability of failure. If ten red-teamers spend two weeks trying to jailbreak a model into providing dangerous synthesis instructions and none of them succeed, this is evidence the model is not trivially exploitable by that group, with those techniques, in that time. It is not evidence that a more persistent attacker, with more domain knowledge or more attempts, would also fail. Absence of evidence in an adversarial search is a much weaker signal than absence of evidence in a random sample, and treating the two as equivalent is the single most common overstatement I see in safety reporting.
This matters practically because extreme-risk evaluation is used to inform real decisions: whether to release a model, whether to restrict access, whether to add guardrails. If the underlying test only measures resistance to a specific, finite adversarial effort, then decisions built on top of it inherit that narrowness. A model can pass every red-team exercise it was given and still be exploitable by a technique nobody on the team thought to try, simply because red-teaming samples from the space of possible attacks rather than covering it.
A worked example: what the numbers actually mean
Suppose a red team of eight people spends eighty hours total probing a model for uplift in a specific harmful domain, and they manage to extract a genuinely useful piece of dangerous information in three of two hundred attempted conversations. Two ways of reporting this are common. One is 'the model resisted 98.5 percent of attempts', which sounds reassuring. The other is 'the model failed under adversarial pressure in a small but nonzero number of cases', which sounds like a red flag. Both descriptions are accurate; neither tells you the thing you actually want to know, which is how easy it would be for a motivated attacker outside the test, with unlimited time and creative prompting, to reliably reproduce that three-in-two-hundred success rate or better.
Here is the intuition that makes this concrete. Imagine the space of possible prompts as an enormous, mostly unmapped territory, and the model's harmful behaviour as scattered pockets within it. Two hundred attempts is a small number of flashlight beams sweeping across that territory. Finding three pockets tells you pockets exist and gives a rough sense of their density near where you looked. It does not tell you whether there is a much larger, easier-to-find pocket five metres to the left that nobody happened to shine a light on. A determined attacker who knows the terrain, perhaps because they have domain expertise the red team lacked, could find that pocket in minutes.
This is why serious extreme-risk evaluation increasingly separates two questions that red-teaming conflates. The first is elicitation difficulty: given a known harmful capability exists somewhere in the model, how much effort does it take to surface it. The second is capability presence: does the underlying capability exist at all, regardless of whether current prompting techniques can reach it. A model might resist every red-team attempt today and still possess the latent capability, waiting for a better jailbreak technique published six months later. Good evaluation tries to estimate capability presence directly, through techniques like fine-tuning the model on adjacent tasks to see what latent knowledge surfaces, rather than relying solely on whether today's attackers happened to find the door.
What red-teaming is genuinely good at, and where it needs support
None of this means red-teaming is worthless; it means it needs to be understood as one instrument with a specific job, not a general-purpose safety certificate. It is genuinely good at surfacing unknown failure modes that structured benchmarks miss entirely, because human red-teamers bring creativity, domain knowledge and adversarial intent that no fixed test set can anticipate. It is good at stress-testing specific guardrails that were added in response to previous incidents, checking whether a patch actually closed the gap or just moved it. It is also good at producing qualitative case studies that help engineering teams understand the shape of a vulnerability, which is often more actionable than a single aggregate score.
Where it needs support is anywhere a decision requires a quantitative bound rather than a qualitative signal. This is where structured capability evaluations earn their place: held-out benchmarks built specifically to probe dangerous capabilities under controlled conditions, with clearly defined success criteria, run at scale rather than by a small team over a limited window. These evaluations should be leakage-aware, meaning the test items were not present in training data and were not previously published in a form the model could have memorised, otherwise a high pass rate on a safety benchmark tells you about memorisation rather than genuine capability. They should also report uncertainty honestly: a sample of two hundred adversarial attempts carries meaningful statistical noise, and a single point estimate without a confidence interval invites false precision.
The practical takeaway is to treat red-teaming reports as a lower bound on risk, never an upper bound. If red-teamers found a serious failure, take it seriously; that is a real, verified problem. If they found nothing, resist the temptation to read that as proof of safety, and ask what fraction of the plausible attack space was actually searched, by whom, with what expertise, and for how long. Pair adversarial search with structured, leakage-controlled capability testing, and be explicit in any report about which question each method answers. Extreme-risk evaluation is too consequential to lean on a method whose main strength is finding problems, when the claim being made is that none exist.