In this tutorial, we built a complete production-style pipeline for revealing and redacting personally identifiable information using OpenAI privacy filter. We start by setting up the environment and loading a token classification model that defines multiple categories of sensitive data, including names, emails, phone numbers, addresses, and secrets. We then design helper functions to normalize the labels, extract structured extensions, and convert the raw model output into usable formats. From there, we implement a configurable redaction system that allows us to replace sensitive entities with meaningful placeholders, preserving privacy and providing context clarity. Throughout the process, we test the pipeline on formatted examples, convert the output into structured data frames, and prepare the system for batch processing and real-world use.
!pip install -q -U transformers accelerate torch pandas matplotlib huggingface_hub
import os, re, json, time, textwrap, warnings
from pathlib import Path
from collections import Counter
import pandas as pd
import matplotlib.pyplot as plt
import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
warnings.filterwarnings("ignore")
MODEL_ID = "openai/privacy-filter"
OUT_DIR = Path("/content/privacy_filter_outputs")
OUT_DIR.mkdir(parents=True, exist_ok=True)
device = 0 if torch.cuda.is_available() else -1
torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
print("Device:", "GPU" if torch.cuda.is_available() else "CPU")
print("Torch dtype:", torch_dtype)
print("Model:", MODEL_ID)We install all required libraries and set up the pipeline runtime environment. We configure the device selection and configure the paths to store the output. We also print out the order details to make sure everything is ready before uploading the form.
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForTokenClassification.from_pretrained(
MODEL_ID,
torch_dtype=torch_dtype,
device_map="auto" if torch.cuda.is_available() else None
)
classifier = pipeline(
task="token-classification",
model=model,
tokenizer=tokenizer,
aggregation_strategy="simple",
device=device if not torch.cuda.is_available() else None
)
LABEL_MASKS = {
"account_number": "[ACCOUNT_NUMBER]",
"private_address": "[PRIVATE_ADDRESS]",
"private_email": "[PRIVATE_EMAIL]",
"private_person": "[PRIVATE_PERSON]",
"private_phone": "[PRIVATE_PHONE]",
"private_url": "[PRIVATE_URL]",
"private_date": "[PRIVATE_DATE]",
"secret": "[SECRET]"
}
We install all required libraries and set up the pipeline runtime environment. We configure the device selection and configure the paths to store the output. We also print out the order details to make sure everything is ready before uploading the form.
def normalize_label(label):
label = label.replace("B-", "").replace("I-", "").replace("E-", "").replace("S-", "")
return label.strip()
def detect_pii(text):
raw = classifier(text)
spans = []
for item in raw:
label = normalize_label(item.get("entity_group", item.get("entity", "")))
if label == "O" or not label:
continue
spans.append({
"label": label,
"score": float(item["score"]),
"text": item["word"],
"start": int(item["start"]),
"end": int(item["end"])
})
spans = sorted(spans, key=lambda x: (x["start"], x["end"]))
return spans
def redact_text(text, spans, min_score=0.50, mode="typed"):
filtered = [s for s in spans if s["score"] andgt;= min_score]
filtered = sorted(filtered, key=lambda x: x["start"], reverse=True)
redacted = text
for span in filtered:
replacement = LABEL_MASKS.get(span["label"], "[PII]") if mode == "typed" else "[REDACTED]"
redacted = redacted[:span["start"]] + replacement + redacted[span["end"]:]
return redacted
def privacy_report(text, min_score=0.50):
spans = detect_pii(text)
redacted = redact_text(text, spans, min_score=min_score)
return {
"original_text": text,
"redacted_text": redacted,
"span_count": len([s for s in spans if s["score"] andgt;= min_score]),
"spans": [s for s in spans if s["score"] andgt;= min_score]
}We define auxiliary functions to normalize labels and extract personally identifiable information (PII) extensions from model predictions. We implement a redaction function that replaces sensitive parts based on confidence bounds. We combine everything into one reporting function that returns structured output.
sample_texts = [
"My name is Alice Smith and my email is [email protected]. Call me at +1 415 555 0189.",
"Patient Rohan Mehta visited on 2025-04-11 and lives at 221B Baker Street, London.",
"Use API key sk-test-51HxYzDemoSecret987 and send the invoice to [email protected].",
"The public website is https://example.com, but Jane Doe's private portal is https://jane-private.example.net.",
"Account number 123456789012 was linked to Ahmed Khan on 12 March 2024.",
"This sentence has no private information and should mostly remain unchanged."
]
reports = []
for i, text in enumerate(sample_texts, 1):
report = privacy_report(text, min_score=0.50)
report["example_id"] = i
reports.append(report)
for r in reports:
print("\n" + "=" * 100)
print("Example:", r["example_id"])
print("Original:", r["original_text"])
print("Redacted:", r["redacted_text"])
print("Detected spans:")
print(json.dumps(r["spans"], indent=2, ensure_ascii=False))
rows = []
for r in reports:
for s in r["spans"]:
rows.append({
"example_id": r["example_id"],
"label": s["label"],
"score": s["score"],
"detected_text": s["text"],
"start": s["start"],
"end": s["end"],
"original_text": r["original_text"],
"redacted_text": r["redacted_text"]
})
df = pd.DataFrame(rows)
display(df)We create sample inputs and run them through the pipeline to test detection and refinement. We collect the structured results and print the original and revised text for comparison. We also convert the output into a data frame for easier analysis.
json_path = OUT_DIR / "privacy_filter_reports.json"
csv_path = OUT_DIR / "privacy_filter_spans.csv"
with open(json_path, "w", encoding="utf-8") as f:
json.dump(reports, f, indent=2, ensure_ascii=False)
df.to_csv(csv_path, index=False)
print("\nSaved JSON:", json_path)
print("Saved CSV:", csv_path)
if len(df):
label_counts = df["label"].value_counts()
plt.figure(figsize=(10, 5))
label_counts.plot(kind="bar")
plt.title("Detected PII Categories")
plt.xlabel("PII Category")
plt.ylabel("Detected Span Count")
plt.xticks(rotation=35, ha="right")
plt.tight_layout()
plt.show()
plt.figure(figsize=(10, 5))
df["score"].plot(kind="hist", bins=10)
plt.title("Detection Confidence Distribution")
plt.xlabel("Confidence Score")
plt.ylabel("Frequency")
plt.tight_layout()
plt.show()
def compare_thresholds(text, thresholds=(0.30, 0.50, 0.70, 0.90)):
spans = detect_pii(text)
results = []
for threshold in thresholds:
kept = [s for s in spans if s["score"] andgt;= threshold]
results.append({
"threshold": threshold,
"span_count": len(kept),
"redacted_text": redact_text(text, spans, min_score=threshold)
})
return pd.DataFrame(results)
threshold_demo = compare_thresholds(sample_texts[0])
display(threshold_demo)We save the processed output in JSON and CSV formats for persistence and reuse. We visualize the detected PII categories and confidence distributions using plots. We also analyze how changing the boundaries affects detection and redaction behavior.
long_document = """
Customer Support Transcript:
Agent: Hello, may I confirm your name?
Customer: My name is PSP.
Agent: Thanks. Could you confirm your email?
Customer: [email protected].
Agent: And your phone number?
Customer: +91 xxxxx xxxxx.
Agent: Your service address is 45 MG Road, Bengaluru, Karnataka.
Customer: Yes. Also, my backup email is [email protected].
Agent: Please do not share passwords or OTPs.
Customer: The temporary token I received is ghp_demoSecretToken123456.
"""
long_report = privacy_report(long_document, min_score=0.50)
print("\nLONG DOCUMENT REDACTION")
print("=" * 100)
print(long_report["redacted_text"])
print("\nStructured spans:")
print(json.dumps(long_report["spans"], indent=2, ensure_ascii=False))
def pii_audit_table(texts, min_score=0.50):
audit_rows = []
for idx, text in enumerate(texts, 1):
result = privacy_report(text, min_score=min_score)
labels = Counter([s["label"] for s in result["spans"]])
audit_rows.append({
"id": idx,
"original_chars": len(text),
"redacted_chars": len(result["redacted_text"]),
"span_count": result["span_count"],
"labels_found": dict(labels),
"redacted_text": result["redacted_text"]
})
return pd.DataFrame(audit_rows)
audit_df = pii_audit_table(sample_texts + [long_document], min_score=0.50)
display(audit_df)
audit_path = OUT_DIR / "privacy_filter_audit.csv"
audit_df.to_csv(audit_path, index=False)
print("Saved audit CSV:", audit_path)
custom_text = input("\nEnter your own text for PII redaction, or press Enter to skip:\n")
if custom_text.strip():
custom_report = privacy_report(custom_text, min_score=0.50)
print("\nOriginal:")
print(custom_report["original_text"])
print("\nRedacted:")
print(custom_report["redacted_text"])
print("\nSpans:")
print(json.dumps(custom_report["spans"], indent=2, ensure_ascii=False))
else:
print("Skipped custom input.")
print("\nTutorial complete.")We test the pipeline on a longer, real-world document to evaluate robustness. We create an audit-style summary detailing the numbers and categories of PII detected. We also allow custom user input so we can trigger the privacy filter interactively.
In conclusion, we have developed a robust and scalable workflow for privacy filtering that goes beyond simple detection. We systematically evaluated model predictions, applied confidence thresholds, and compared different refinement strategies to understand their impact. We also created structured reports, visual discovery patterns, and exported the results in JSON and CSV formats for final auditing and integration. This approach allows us to build reliable privacy safeguards into data pipelines, ensuring sensitive information is consistently identified and handled responsibly while maintaining the ease of use of the underlying data.
verify Full codes here. Also, feel free to follow us on twitter Don’t forget to join us 130k+ ml SubReddit And subscribe to Our newsletter. I am waiting! Are you on telegram? Now you can join us on Telegram too.
Do you need to partner with us to promote your GitHub Repo page, face hug page, product release, webinar, etc.? Contact us
