PII redaction benchmark: Shadow vs Presidio, GLiNER, OpenAI Privacy Filter, LLM Guard
Shadow is an open-source privacy layer that masks PII and secrets in LLM prompts with reversible fakes. This page measures how much sensitive data it and five other open-source tools let through, how much harmless text they destroy, and how fast they are, on three labelled datasets. Everything here is reproducible from the bench/ folder.
Why leak rate is the headline number
Section titled “Why leak rate is the headline number”When a prompt goes to a hosted model, a missed value is a value disclosed. So the main metric counts sensitive values still readable in the output, after normalising case, spacing and punctuation, rather than span-level F1, which rewards finding most of a value. A tool that masks “Jane” but leaves “Okafor” in the text has leaked a surname; F1 calls that a partial hit.
Leak rate alone would reward masking everything, so the benchmark also measures:
- Public names changed: labelled names that should stay readable (Microsoft, Berlin, Taylor Swift). Masking them makes answers worse.
- Text kept: ordinary words outside any label that survive. Low values mean the model gets a mangled prompt.
- Precision, recall, F1: span detection, a gold span counts as found when at least half of it is covered.
- Round trip: for tools that can reverse their masking, whether unmasking gives back the original exactly.
- Latency: median time per document on one CPU, after warm-up.
Held-out test set
Section titled “Held-out test set”1,500 documents, 7,800 labelled sensitive values, ten locales and eleven scenarios. Generated with a different seed from the development set and never used for tuning. Lower is better for the first three columns.
| Tool | Leak rate ↓ | Partial name leak ↓ | Public names changed ↓ | Text kept ↑ | Precision | Recall | F1 | Round trip | p50 latency |
|---|---|---|---|---|---|---|---|---|---|
| Shadow + open-weight NER sidecar | 0.08% | 0.45% | 2.06% | 86.48% | 78.53% | 99.62% | 87.83 | 99.27% | 1003 ms |
| Shadow + Presidio detector | 1.35% | 4.87% | 0% | 94.78% | 95.64% | 97.77% | 96.69 | 100% | 22.8 ms |
| Shadow (built-in rules) | 3.08% | 9.8% | 0% | 98.13% | 99.97% | 96.46% | 98.19 | 100% | 7.5 ms |
| OpenAI Privacy Filter | 11.01% | 4.57% | 0.61% | 93.12% | 86.55% | 85.87% | 86.21 | not reversible | 227 ms |
| GLiNER multi PII | 13.28% | 1.36% | 49.28% | 88.88% | 74.6% | 86.58% | 80.14 | not reversible | 405 ms |
| Microsoft Presidio | 20.74% | 22.76% | 26.01% | 90.21% | 79.39% | 72.64% | 75.87 | not reversible | 8.8 ms |
| LLM Guard (Anonymize) | 29.26% | 15.73% | 2.29% | 97.87% | n/a | n/a | n/a | 100% | 110 ms |
| scrubadub | 67.38% | 92.36% | 0% | 98.14% | 100% | 34.37% | 51.16 | not reversible | 0.4 ms |
What stands out:
- Structured identifiers are solved by rules. Shadow leaked no emails, phone numbers, IBANs, card numbers, national IDs, IP addresses, URLs, hostnames, usernames or secrets. ML-only tools miss many of these: GLiNER leaked 89% of national IDs and 80% of URLs; Privacy Filter leaked 59% of IP addresses and 69% of usernames; Presidio leaked 74% of secrets.
- Names are where tools differ. Shadow’s rules still leak 21.6% of company names and 5.6% of person names, mostly names in free text with no surrounding cue. The NER sidecar closes that gap (1.1% and 0%), at the cost of precision and some over-masking.
- GLiNER over-masks. It changed 49% of public names, so prompts about real companies or places lose their meaning.
- LLM Guard reports no spans, so precision and recall are not available for it.
Leak rate by type of value (held-out)
Section titled “Leak rate by type of value (held-out)”| Value | Shadow | Shadow + Presidio | Shadow + NER sidecar | Presidio | Privacy Filter | GLiNER | LLM Guard | scrubadub |
|---|---|---|---|---|---|---|---|---|
| address | 2.15% | 0.59% | 0% | 14.48% | 3.13% | 16.83% | 92.56% | 99.61% |
| credit card | 0% | 0% | 0% | 2.8% | 0% | 0% | 18.18% | 23.78% |
| date of birth | 0% | 0% | 0% | 0% | 0% | 0% | 100% | 100% |
| 0% | 0% | 0% | 0% | 0.28% | 5.8% | 0% | 0% | |
| hostname | 0% | 0% | 0% | 0% | 1.42% | 46.81% | 0% | 0% |
| iban | 0% | 0% | 0% | 6.23% | 0.33% | 0% | 9.18% | 98.69% |
| ip | 0% | 0% | 0% | 0% | 58.87% | 21.99% | 0% | 100% |
| national id | 0% | 0% | 0% | 22.92% | 0% | 88.89% | 88.19% | 88.89% |
| org | 21.61% | 10.62% | 1.1% | 61.17% | 55.68% | 3.66% | 81.87% | 100% |
| person | 5.58% | 2.21% | 0% | 22.86% | 4.02% | 0.1% | 1.31% | 100% |
| phone | 0% | 0% | 0% | 3.68% | 0.49% | 13.82% | 16.19% | 54.7% |
| secret | 0% | 0% | 0% | 74.17% | 0% | 1.28% | 51.15% | 98.98% |
| url | 0% | 0% | 0% | 0% | 30.83% | 80.45% | 100% | 0% |
| username | 0% | 0% | 0% | 89.43% | 69.04% | 56.76% | 85.26% | 100% |
Leak rate by kind of text (held-out)
Section titled “Leak rate by kind of text (held-out)”| Scenario | Shadow | Shadow + Presidio | Shadow + NER sidecar | Presidio | Privacy Filter | GLiNER | LLM Guard | scrubadub |
|---|---|---|---|---|---|---|---|---|
| chat request | 7.28% | 2.43% | 0.13% | 21.33% | 16.99% | 1.66% | 32.69% | 76.12% |
| code | 0% | 0% | 0% | 49.28% | 5.44% | 26.56% | 56.32% | 59.36% |
| CSV export | 0.38% | 0.14% | 0% | 8.73% | 0.58% | 4.99% | 9.88% | 52.09% |
| DevOps log | 0% | 0% | 0% | 40.9% | 49.05% | 49.29% | 54.49% | 66.67% |
| HR record | 0.42% | 0.14% | 0% | 10.56% | 0.14% | 26.94% | 57.22% | 89.44% |
| meeting notes | 15.93% | 8.14% | 0.18% | 33.98% | 18.76% | 0.18% | 17.52% | 80% |
| multilingual | 1.46% | 0.49% | 0% | 12.34% | 3.41% | 1.14% | 20.13% | 68.18% |
| payment | 9.01% | 4.8% | 0.9% | 34.23% | 15.32% | 3.3% | 37.24% | 100% |
| email signature | 6.56% | 2.6% | 0.15% | 18.02% | 12.82% | 18.78% | 31.76% | 72.37% |
| support ticket | 0% | 0% | 0% | 6.82% | 0.35% | 0% | 6.99% | 55.24% |
Third-party test set: Gretel synthetic PII finance
Section titled “Third-party test set: Gretel synthetic PII finance”1,000 documents sampled from the test split of gretelai/synthetic_pii_finance_multilingual (Apache-2.0), with 5,077 sensitive values in seven languages: loan agreements, SWIFT messages, customer records, privacy notices and similar financial documents. It was written by a third party and never used to tune Shadow; detector rules were developed only on samples of its separate train split. Gretel does not label public names, so over-masking is not measured here.
This is the honest picture of out-of-distribution performance:
- Every tool leaks more on these documents than on the synthetic sets. Long legal and financial prose, company names in many languages and names without cues are harder for everyone.
- GLiNER alone beats Shadow’s rules alone on overall leak rate (17.61% vs 31%), driven by company names (6% vs 47%) and person names. It pays for that with 35% precision: almost two in three things it masks are not sensitive. It also took 2.3 seconds per document on CPU.
- Shadow’s rules stay ahead on identifiers (phones 4%, IP 4.5%, IBAN 5%) and keep the most text intact (98.8%) with 89% precision.
- Part of Shadow’s company-name leak is deliberate: its public allowlist keeps well-known company names readable, and Gretel labels every company as sensitive.
Gretel leak rate by language
Section titled “Gretel leak rate by language”Development set
Section titled “Development set”The development set (1,500 documents, 7,637 values, seed 20261031) is what Shadow’s rules were tuned against, so its numbers flatter Shadow and are shown only for completeness. They match the held-out set closely (Shadow rules 3.16%, + Presidio 1.31%, + NER sidecar 0.1%; Privacy Filter 11.27%, GLiNER 13.75%, Presidio 20.65%, LLM Guard 29.8%, scrubadub 68.08%), which suggests the rules are not overfitted to individual examples, though they share the generator’s style.
Methodology
Section titled “Methodology”Datasets
Section titled “Datasets”| Dataset | Documents | Sensitive values | Source | Use |
|---|---|---|---|---|
heldout-v1 | 1,500 | 7,800 | Shadow generator, seed 7 | test, never tuned on |
gretel-finance-1000 | 1,000 | 5,077 | Gretel.ai, test split, Apache-2.0 | test, never tuned on |
synthetic-v1 | 1,500 | 7,637 | Shadow generator, seed 20261031 | development |
The Shadow generator writes original text with Faker in ten locales and eleven scenarios: chat requests to an assistant, support tickets, CSV exports, HR records, DevOps logs, source code, meeting notes, email signatures, payments and multilingual messages. Every sensitive value is labelled, and so are public names a good tool should leave alone. No real personal data is used anywhere.
Gretel labels are mapped onto the benchmark’s: name, first_name and last_name to person, company to org, street_address to address, ssn, passport_number and driver_license_number to national ID, password and api_key to secret, and so on. Labels with no counterpart (times, plain dates, internal customer IDs, BIC codes) become ignore spans: masking them is not a false positive and leaving them is not a leak. The full mapping is in bench/datasets/gretel.py.
Tools and configuration
Section titled “Tools and configuration”| Tool | Version | Configuration |
|---|---|---|
| Shadow (built-in rules) | main at publication | default config: regex, checksums, dictionaries, names, fields, addresses, public allowlist |
| Shadow + Presidio detector | Presidio Analyzer 2.2.362 container | Shadow’s built-in rules plus Presidio over HTTP |
| Shadow + open-weight NER sidecar | ml/serve_hf.py --backend ensemble | built-in rules plus OpenAI Privacy Filter and GLiNER multi PII v1 |
| Microsoft Presidio | presidio-analyzer 2.2.358, spaCy en_core_web_lg | default recognisers, replace operator |
| OpenAI Privacy Filter | openai/privacy-filter | sidecar --backend privacy-filter, all predicted spans masked |
| GLiNER multi PII | urchade/gliner_multi_pii-v1, gliner 0.2.21 | PII labels, threshold 0.5 |
| LLM Guard | llm-guard 0.3.16 | Anonymize scanner with its default vault, Deanonymize for round trip |
| scrubadub | 2.0.1 | default detectors |
Every tool ran with its defaults or its documented recommended settings. No tool was given the gold labels. Each tool was run on the same text and scored by the same code.
Hardware
Section titled “Hardware”One Apple M3 (8 cores), CPU only, PyTorch 2.15 and transformers 4.51. Apple’s GPU backend crashed on this macOS version, so all models ran on CPU; on a GPU the model-based tools, and Shadow with its sidecar, are many times faster. Latency is the median per document after warm-up, measured inside each tool’s process and including Shadow’s unmask check.
Limitations
Section titled “Limitations”- The synthetic sets come from Shadow’s own generator. Their formats and phrasing are the ones Shadow’s rules were written for, which favours Shadow. Treat the Gretel numbers as the fairer comparison.
- Gretel’s documents are synthetic too, generated by an LLM, and only financial.
- Leak rate checks whether the exact value survives. A tool that changes one character of a value is counted as having masked it.
- Shadow + NER sidecar does not round-trip every sample (99.27% held-out): an ML span occasionally overlaps text a fake collides with. Built-in rules alone round-trip 100%.
- Commercial services (Private AI, Nightfall, AWS Comprehend, Google Sensitive Data Protection, Azure AI Language) are not included because their terms or pricing make public, reproducible runs impractical. See the comparison pages for what their documentation says.
Reproduce
Section titled “Reproduce”git clone https://github.com/Tiny-Bubble-Company/shadow && cd shadow/benchpython -m venv .venv && . .venv/bin/activatepip install presidio-analyzer presidio-anonymizer gliner scrubadub llm-guard faker huggingface_hub pyarrowpython -m spacy download en_core_web_lg
python run.py --data datasets/heldout-v1.jsonl --out results-heldout \ --tools shadow,presidio,gliner,scrubadub,llm_guardpython report.py results-heldoutshadow+presidio needs a Presidio Analyzer on 127.0.0.1:5002; shadow+ner and privacy_filter need the open-weight sidecar on ports 8878 and 8879. The bench README has the details, and summary.json for each run is committed next to the code.
Related
Section titled “Related”- How Shadow detects sensitive data
- Shadow vs Presidio, vs GLiNER, vs OpenAI Privacy Filter, vs LLM Guard
- Open-source PII redaction tools compared
Frequently asked questions
Which open-source tool leaks the least PII in LLM prompts?
On the held-out test set, Shadow with its open-weight NER sidecar leaked 0.08% of sensitive values, Shadow with the Presidio detector 1.35%, and Shadow's built-in rules alone 3.08%. The best standalone tools were OpenAI Privacy Filter (11.01%), GLiNER multi PII (13.28%) and Microsoft Presidio (20.74%). On the third-party Gretel finance set every tool leaks more; see the Gretel section for those numbers.
Is Shadow more accurate than Microsoft Presidio?
On all three datasets in this benchmark Shadow's built-in rules leak fewer sensitive values than Presidio's default analyzer (3.08% vs 20.74% held-out, 31% vs 43.39% on Gretel), with higher precision and no public names changed. Shadow can also run Presidio as an extra detector, which lowers its leak rate further.
How fast is Shadow compared with other PII tools?
Shadow's built-in detectors took 7.5 ms per sample at the median on one Apple M3 CPU, including masking and the round-trip check. Presidio took about 9 ms, LLM Guard about 110 ms, OpenAI Privacy Filter about 230 ms and GLiNER about 400 ms on the same CPU. Adding the open-weight NER sidecar makes Shadow about 1 second per sample on CPU; a GPU cuts that sharply.
Are these benchmark results biased towards Shadow?
Partly, and the page says where. The synthetic test sets come from Shadow's own generator, and its rules were tuned on a separate development set from the same generator, so it has a home advantage there. That is why the page also reports a third-party dataset (Gretel synthetic PII finance, Apache-2.0) that was never used for tuning. On that set, GLiNER alone leaks less than Shadow's rules alone.
Can I reproduce the benchmark?
Yes. The datasets, label mappings, runner and report script are in the bench folder of the Shadow repository on GitHub, under Apache-2.0. One command runs every tool and writes the tables on this page.
Which tools can restore the original values after the LLM answers?
Only Shadow and LLM Guard's Anonymize/Deanonymize pair round-trip in this benchmark. Shadow restored the exact original text in 100% of samples with its built-in rules. Presidio, GLiNER, OpenAI Privacy Filter and scrubadub replace values one way.