blog/measuring-llm-guardrails-false-positives-attack-success-budget-hero
← Back to blog

Measuring an LLM Guardrail: Error Costs, an Attack Budget in CI and Where Cheap Classifiers Fail

Technology•17 November 2026•TREEO Indonesia•5 min read

A guardrail that nobody has measured is a belief, not a control (the same discipline applies to measuring RAG hallucination claim by claim). It passes the three demo prompts the team tried and fails silently on everything else. The 2026 literature on guardrail evaluation is thin, with few independent benchmarks, but it converges on a method that any team can apply with a spreadsheet and a test runner. This guide sets out that method and then shows, with published numbers, why the cheap shortcuts go wrong.

Guardrail measurement in numbers

4

outcomes to count for every guard decision: true positive, false positive, true negative, false negative

0.05

attack-success-rate ceiling used as the example gate in one engineering guide

46.34%

input-gate bypass rate reported for one tutoring system, with a 0.00% false-positive rate

0.004

F1 reported for one injection classifier on a shifted-source test set, after 0.721 in-distribution

Start with a labelled corpus and four counts

The core method, in one engineering guide, is to build a labelled corpus, run each item through the guard, and count four outcomes. A true positive is an attack blocked, a false positive is a legitimate request blocked, a true negative is a legitimate request allowed, and a false negative is an attack allowed. From these come precision, recall, F1 and the false-positive rate.

The guide's central point is that the two errors have opposite costs. A false positive frustrates legitimate users and erodes trust. A false negative is a safety incident. So a single accuracy number hides what matters. The recommended operating point for most safety guardrails is high recall while holding the false-positive rate inside a budget the product can tolerate, and tracking both over time. A drop in recall after a dependency upgrade is flagged as an early warning that a guardrail quietly broke.

Corpus hygiene matters as much as the metrics. The same guide advises a versioned golden corpus, updated in the same commit as any policy change, because a policy edit can create two regressions at once: blocking what used to pass, or passing what used to be blocked. When an incident happens, the failing case goes into the corpus so that exact attack stays blocked.

Turn the attack rate into a build gate

For red-team testing, the guide mutates a base set of known attacks and defines the attack success rate (ASR) as failures divided by the number of mutated attacks. The build is gated on a threshold, with `assert asr <= 0.05` as its example. That value is an illustration, not a standard; the right budget depends on the harm if an attack lands. The guide's line worth remembering is that a red-team suite which only prints results is documentation, while one that fails the build is a guardrail. It also says to re-run the suite whenever the model version changes, since a new model can reopen attacks the old one resisted.

The same guide adds a decision many teams skip: whether each guard fails open or fails closed when it errors. Fail-closed blocks the request when the guardrail itself breaks, the safer default for regulated or high-stakes uses at the cost of availability. Fail-open keeps the service up and suits only low-risk checks. Either way, test the failure path so a swallowed exception never silently passes traffic.

A worked example, and its honest reading

An arXiv paper on an educational tutoring assistant gives one of the few complete operating points in the sources. It builds 480 synthetic queries with LLM assistance and human review: 369 injections across six families (instruction override, role hijacking, format bypass, multi-turn escalation, prompt exfiltration and obfuscated or encoded payloads) and 111 benign student questions. The input gate reports 46.34% bypass, a 0.00% false-positive rate and 2.50 ms average latency.

The authors decline to call that "security". Nearly half of the attacks pass the gate. Their response-level audit, with the gate combined with a pedagogical system prompt and a content-filtered provider, finds a 0% critical bypass rate. The lesson is to report per layer as well as end to end: a high single-layer miss rate is a failing layer, even if the whole system holds. The paper also uses stratified bootstrap confidence intervals and paired McNemar tests to compare configurations, rather than trusting one run. The six-family taxonomy is a handy way to stratify a corpus.

Cheap classifiers: strong inside the data, brittle outside it

A benchmark preprint compared CPU classifiers (for example a TF-IDF SVM) with GPU models across nine jailbreak sources in three regimes: in-distribution (D1), out-of-distribution (D2) and adversarially obfuscated (D3). On D1 the best CPU classifier matched the best transformer at roughly one-fifth of the deployment cost; the study puts the SVM at about $0.17 per million requests. On D2 the CPU classifiers failed through confident miscalibration, producing high-confidence false negatives that bypass escalation entirely. The off-the-shelf guards it tested were inconsistent across regimes:

ModelD1 F1D2 F1D3 F1
DeBERTa-v3 prompt-injection classifier0.7210.0040.695
PromptGuard-86M0.7030.9840.085
WildGuard-7B0.7090.8990.628

The paper warns that the 0.984 is an artefact: that model labelled all 100 benign D2 samples unsafe, so the score is inflated by class skew and says nothing good about it. PromptGuard also collapsed on obfuscated text (0.085), which the authors attribute to a tokeniser trained on clean English. The practical reading: a score on data like your training set is an upper bound, not a forecast. Build a corpus slice from different sources and one with obfuscation, and report each separately.

Perplexity-based detection and its limits

Another proposed approach scores how "surprising" a prompt is to a language model. Adversarial suffixes produced by optimisation tend to look statistically odd. One source describes proposals to augment plain perplexity with features such as prompt length and token-frequency statistics and feed them to gradient-boosted trees or density estimators. The same source reports a sequence-level AUROC of 1.00 for adversarial prompts. That figure is the source's own attribution and could not be independently checked, and the source does not say whether it covers fluent, human-written jailbreaks as well as optimised suffixes.

The limits are documented too. One source notes that reference-model perplexity is unreliable on noisy, foreign or garbled text, extreme values can reflect model blind spots instead of true anomalies, and a source on detecting machine-written text more generally notes that accuracy falls on shorter texts. The sources found no head-to-head benchmark comparing perplexity filtering with classifier ensembles, so treat it as one signal among several, to be measured on your own corpus like any other.

For multilingual traffic the evidence is weaker still. A secondary summary of SGToxicGuard reports guardrails with elevated false-positive rates in low-resource languages, but no source gives per-language tables. Count errors per language variety yourself.

Frequently Asked Questions

Accuracy hides the asymmetry. Blocking a safe request and allowing an attack have different costs, so report both error rates per guard.

Turn climate goals into a verified portfolio

TREEO connects the full carbon cycle — eligibility, simulation, real-time monitoring and registry-ready reporting — combining expert consulting with dMRV technology so the evidence exists before anyone asks for it.