Issue #3, Sunday, September 27, 2026

Hey everyone, I decided to spend a little time on the theme of the newsletter, so I hope you like it. Last week I graded a paper whose code link was dead. This week I grade my own tool with the same four questions. Fair is fair. Numbers at the bottom, as always.

But first, the problem the tool is built for.

Rules fail silently

A detection rule that stops working does not throw an error. It just stops matching. Your SIEM shows no alert, and "no alert" looks exactly like "no attack."

That is why rule decay is dangerous. Nothing tells you. You find out during an incident, when a rule you trusted did not fire.

"Detection drift" is usually treated as one problem. It is at least seven.

The seven types

1. Schema drift. A field is renamed or moved in the pipeline. The rule still runs, but it looks at a field that no longer holds the data.

2. Translation drift. The tool that converts your Sigma rule into a SIEM query changes. The same rule now produces a different query.

3. Rule decay (zero-fire rot). A data source goes dark or enrichment breaks. The rule can never fire again.

4. Threshold drift. "More than 20 failed logins in 5 minutes" was right when you had 500 users. You now have 5,000.

5. Suppression creep. Each false positive gets a filter. After three years of filters, the rule excludes the attack it was written for.

6. ATT&CK drift. The framework changes (techniques move, split, or retire), and the text of a technique changes. Your rule still maps to the old meaning.

7. Governance drift. The author leaves. The severity no longer fits. Nobody owns the rule.

Each type has a different signal and a different fix. A tool that says it solves "drift" should tell you which ones.

What AI can catch today, and what it cannot

This is what detect-forge does today. It is free and open source: pip install detect-forge.

detect-forge stale addresses type 6. It checks two things. Timestamp drift: was the ATT&CK technique updated after your rule was last changed? Semantic drift: did the meaning of the technique change? It compares the rule's intent with the current technique text, using embeddings. When it finds drift, an LLM proposes a change to the rule. You approve every change. Nothing merges on its own.

detect-forge backtest addresses types 3 and 4. It runs your rule against labeled attack data and reports precision, recall, and F1. A rule with zero recall on data where the attack is present has decayed.

What it does not cover: types 1 and 2 happen inside your SIEM and pipeline, and a tool that reads rule files cannot see them. Type 5 is a concept, not a feature. Type 7 is an organizational problem, not a technical one.

So the honest answer is: three of the seven types, partly.

Graded with my own rubric

Q1. Can I validate it? Yes, in part. It is free and runs locally on your own rule files.

Q2. Can it be validated? In part. "The rule still catches the attack" is testable only where labeled attack data exists for that technique. For other techniques, the tool can flag drift, but it cannot prove detection.

Q3. Where is the code? Public on GitHub and PyPI. Not public yet: reproducibility manifests, holdout splits, and frozen trial results. Those are next.

Q4. Feasible at scale? It needs no SIEM access, so it is easy to adopt. That is also its limit: it cannot see schema changes in your live SIEM, and a public attack dataset is not your telemetry.

Verdict: LAB-ONLY. It changes when the manifests and holdouts ship, and when I have results from real rule sets, misses included.

Correction

Issue #2 and version 1 of the QAS Rubric said that CICIDS2017 is distributed as flow-summary CSV files. That is wrong. The dataset also includes the raw PCAP files. The correct point is narrower: most reproductions use the CSV files, which cannot supply the four packet-level features that PARD-SSM needs. The verdict on the paper does not change. Rubric version 2 is corrected and dated.

The QAS Rubric is the four questions, the six verdicts, four worked verdicts, and a one-page checklist. New subscribers get it at signup.

Filtered

ALIBI (arXiv 2609.19722): a false "security product" story, added to a malware binary as a section that never runs, made an LLM analyzer call 30 of 35 malicious samples benign. The lesson: text inside the artifact is a claim, not evidence.

Robust Conformal Intrusion Detection (arXiv 2609.19241): a coverage guarantee that survives attacks, if you score only the features that the attacker cannot touch. The cost is 7 to 14 points of clean accuracy. Code is promised on acceptance.

The numbers

Week three. Numbers are all the same so far. But onward and upward.

Before you go

Run detect-forge stale on your rule library and hit reply with what it flags. If you see a drift type that does not fit the seven, reply with that too. I will add it and credit you.

Next Sunday: Spectrum reported that 29.2% of deployed detection rules cannot fire. I grade the report with the same four questions.