My detection models score 100% on everything. That's a red flag, not a feature.
A weekend detection lab, sixteen trained models, perfect metrics on almost every row — and why the most honest number on the dashboard was a 0.0%.
The setup
Last weekend I built a small multi-domain detection lab: network flows, phishing URLs, identity anomalies, malware memory metadata, and LLM prompt abuse. Same origin story as the detection rule backtester — scaffolding recycled from a dead lottery-prediction project — but this time pointed at the other half of the problem: not measuring rules, but running a scoring pipeline end to end. Events come in, models score them, anything above a 0.55 risk threshold opens an alert. A mini-SOC in a box, labelled LAB/DEMO on every page.
I trained the whole zoo — heuristic rules, random forests, a TF-IDF text classifier, voting ensembles; sixteen models across the five domains — and opened the metrics tab. Here is what it showed:
F1: 100%. Precision: 100%. Recall: 100%. False-positive rate: 0%. On nearly every row.
If I were selling this, that screenshot would be the pitch deck. Since what I actually care about is evaluating detection systems, my reaction was the opposite: something is wrong. A perfect detector doesn't exist. A detector that believes it's perfect is a detector nobody has seriously evaluated yet.
Where the 100% comes from
The models are trained and evaluated on synthetic fixtures: two hundred events per domain, generated by my own code. The "injection" prompts contain phrases like ignore all previous instructions; the benign ones are ordinary questions. The phishing URLs have raw IPs and shady TLDs; the clean ones don't. The malicious network flows are loud; the normal ones are quiet.
In other words: I wrote an exam where the questions are the answers. The two classes are separable by construction. Any model — a single regex — scores 100% on this data. The metric isn't measuring the quality of the detector; it's measuring the naivety of the dataset.
That's the first reflex worth building in front of any detection metric: ask what it was measured on before asking how high it is. A vendor quoting "99.9% detection" without describing the evaluation set has told you literally nothing. My dashboard is that vendor slide, self-hosted.
The row that gives the game away: 0.0%
The most instructive number in the table isn't any of the 100%s. It's one row showing 0.0% across every column: the network domain's anomaly detector, an Isolation Forest.
Why zero? Because an Isolation Forest is unsupervised — it doesn't predict "attack / benign", it assigns an outlier score. My evaluation pipeline computes precision and recall as if every model were a supervised classifier. Supervised metrics applied to a model that isn't playing that game produce meaningless zeros — printed with exactly the same confidence as the perfect scores two rows up.
The lesson travels beyond my lab: a dashboard prints numbers, not truths. If the evaluation harness is wired wrong, it produces numbers anyway. The 100%s and the 0.0% on my screen have the same root cause — an evaluation that wasn't designed with the same seriousness as the system it measures.
What honest evaluation looks like
Turning this lab into something defensible is the point of the next posts. The plan:
1. Real, public data. Re-evaluate on recognized benchmarks — CIC-IDS2017 and UNSW-NB15 for network, CIC-MalMem-2022 for malware, public jailbreak corpora for the LLM domain. Prediction: the metrics will fall off a cliff, and the number after the fall is the one worth publishing.
2. Adversarial pressure. A prompt-injection detector that recognizes "ignore all previous instructions" verbatim — what does it do against a paraphrase, a base64 wrapper, an indirect injection? Measure a bypass rate, not just an F1. The backtester post already put a floor under this: keyword rules caught 8.8% of real in-the-wild jailbreaks, and a tuned ensemble still left 30% through.
3. False positives at the operating threshold. My system opens an alert at a score of 0.55. The metric a SOC team lives with isn't an abstract ROC-AUC — it's how many false alerts per day at that exact threshold. Alert fatigue kills more detection programs than missed attacks do.
4. Evaluate the evaluator. Check that every model type is scored with metrics that make sense for it. My Isolation Forest row is what skipping that step looks like.
Why I'm leaving the dashboard up
Companies are wiring LLMs into everything and buying "prompt firewalls" on the strength of demos. The rare skill in 2026 isn't training a classifier — it's being able to prove what a detection system is actually worth, and being honest about what it isn't. This lab is my public training ground for exactly that.
So the 100% table stays, banner attached: demo metrics ≠ production performance. Next post, I feed it real data — and we watch together how far it drops.
The lab (Cyber-XWin) is a personal LAB/DEMO project — five detection domains, FastAPI + scikit-learn, dry-run prevention by default. It is deliberately not a product; it's the system under test for the detection rule backtester.