Last week I told you what this newsletter is: research most of the field never sees, explained, then rebuilt to see if it holds. This is the first one. Numbers at the bottom, as promised.

A paper crossed my desk this spring claiming 98.2% F1 on intrusion detection. I did something unusual with it: I reimplemented the whole model. Here's the strange part. After weeks inside the code, I think the 98.2% is the least interesting number in the paper, and the idea underneath it is one of the more useful things I've seen this year.

First, what a regime is

A regime is a mode a system is operating in. The mode is hidden. Only its symptoms are visible. Weather is the easiest example: you never observe "a cold front," you observe temperature dropping and pressure shifting, and once the front arrives the next few hours behave in predictable ways. A good forecaster doesn't ask "is this reading unusual?" They ask "which regime are we in, and what does it do next?"

A network under attack is the same shape. The attack phase is hidden. Scans, odd logins, process trees, data leaving are its symptoms, and the phases happen in a rough order. In machine learning this is a switching state-space model. The idea that attack stages are hidden states goes back to hidden Markov model research on multi-stage attacks in the early 2000s. What's new is that these models now run fast enough to matter.

The paper that tried it

The paper is PARD-SSM (arXiv 2604.02299): probabilistic cyber-attack regime detection via switching state-space models. Strip the math and the core claim is this: attacks are regimes, not anomalies. Most ML detection asks "is this data point weird?" and hands your analysts an anomaly score with no story attached. A regime model asks a different question: "which phase of an attack are we probably in?" It runs two coupled hidden processes, a discrete one tracking the kill-chain stage (normal, recon, intrusion, exfiltration) and a continuous one tracking network dynamics, where each stage implies different dynamics. The output isn't a weirdness score. It's a posterior over attack stages, updated every second, that can be pushed forward in time: not just "something's wrong" but "this looks like reconnaissance, and the transition structure says intrusion is what usually comes next."

That reframing, from point anomalies to latent campaign phases, is right, and it's durable. It matches how attacks actually unfold and how analysts actually reason. So why do I dismiss the headline number?

Because 98.2% F1 on CICIDS2017 is close to information-free. CICIDS2017 is a 2017 academic capture that the field has been overfitting to for nearly a decade; strong baselines cluster in the high nineties, the paper reports no error bars, and a saturated benchmark can't separate a good model from a well-tuned one. The numbers that actually distinguish this work are the ones nobody headlines: 86.1% stage attribution (when it fires, it tells you which phase, and that's triage information, not just an alarm), an ~8-minute average early-detection margin (alerts before the labeled attack onset, a metric almost nobody in our industry even measures), and under 1.2ms per flow on a plain CPU (no GPU tax, which is the difference between a paper and a deployment).

What rebuilding it taught me

Reimplementing it taught me things the PDF couldn't. Three worth your time:

First, the benchmark can't even compute the paper's own features. Four of the seventeen input dimensions require raw packet captures; CICIDS2017 is distributed as flow-summary CSVs. Anyone "reproducing" this paper from the standard download is quietly substituting proxies. My implementation does it openly and budgets ±0.01 F1 of drift for it. When the dataset can't express the method, the benchmark score is measuring something other than the method.

Second, reproducibility here is worse than it looks. The paper shields its "specific model parameter values" behind a pending patent, and its published code link, as of this writing, returns a 404. So I rebuilt it clean-room from the spec and set explicit acceptance bands (F1 within ±0.010 of the paper, stage attribution within ±0.015). As far as I can verify, my implementation is currently the only public one in existence.

Third, switching models fail in ways papers don't discuss. Mine ships a mode-collapse detector because in practice the model sometimes decides everything is one regime and stops switching. If you evaluate a vendor product built on ideas like these, ask what happens when the regimes collapse. Watch the answer carefully.

What should you actually do with this? Two things, neither of which requires deploying anyone's patent-encumbered model. Steal the framing: next time you review a detection, ask what it believes about attack phase, not just about weirdness. Anomaly scores make analysts do the campaign reasoning; regime posteriors do some of it for them. And steal the metric: start measuring your detections' early-detection margin, how far ahead of impact they fire, not just whether they fire. It's the number that decides whether automation has time to act, and almost nobody tracks it.

The frozen prediction

I haven't run the full reproduction yet. That's deliberate. Here's my prediction, in writing before the experiments: I expect weighted F1 within ±0.010 of the paper's 98.2% on CICIDS2017, and I think stage attribution is the number at real risk of missing its band. Results land in the October 11 issue, hit or miss. That's the standard this letter grades others against, so it applies to me first.

The company has a name

Last week I said I was building an AI security company with zero employees. It is called DetectTrace. The thesis is one sentence: no matter how good the detection rules are, the analyst still has to answer "is this normal for this identity, here, now," and nothing in the alert answers it. DetectTrace is the context layer that does. It sits above whatever detection stack you already run, takes the alert, assembles the context that decides it, and hands back a case with the evidence attached. detect-forge stays free and open source underneath it, keeping the rules honest. More next week, including the first design partner if I have one.

Quantitative AI Security on LinkedIn

Last week I said there was more coming under the name. Here's the first piece. The LinkedIn group I've run for years (3,000+ members) is now Quantitative AI Security. Same four questions, same standard, a place to argue about the papers and the tools with other people who read past the abstract. If you're on LinkedIn, Join Here

Filtered

PARD-SSM (arXiv 2604.02299): the paper itself. Read Section 1 for the regime framing even if you skip the variational machinery; the problem statement is the best-written part.

The "HED Score" follow-up (arXiv 2604.04993): same lead author, one month later, a self-named metric pitched as the successor to ROC/AUC, evaluated on NSL-KDD, a benchmark the field retired years ago. Graded: the instinct is right (time-to-detection deserves a first-class metric), the validation isn't there yet. Notice that even the author's follow-up agrees about which number matters.

lindermanlab/ssm: if regime models made you curious, this is the mature open-source library for state-space models (HMMs, switching LDS, variational inference). The fastest way to build intuition on your own data this weekend.

Cybersecurity_Regime_Change: my reimplementation. Model core, KL-gated predictive alerting, 54 tests, and the reproduction harness with the acceptance bands above. MIT-licensed; issues and pull requests welcome, especially from anyone who's fought CICIDS2017's CSVs before.

The numbers

Week one, as promised. Readers: 40, with zero new and one unsubscribe. GitHub stars: 0. PyPI downloads: 17 last week, 239 since release, and that count includes mirrors and bots, so I am treating it as release noise until it moves week over week. Paying customers: 0. Income: $0. Runway: about 6 months. Baseline last week was 40 readers, 0 stars, $0. Nothing has moved yet. That is the honest read of week one, and it is the point of publishing it.

Before you go

In your SOC, how long is the gap between "something's weird" and "we know which stage of an attack this is"? Minutes? Hours? Never measured it? Hit reply. I read everything, and the answers will shape the reproduction write-up.

P.S. If you maintain detection rules: pip install detect-forge. Free, local, and it tells you which of your rules still catch what they claim. If you'd rather I run it across your estate: jamesbower.com/audit.

---

James Bower. I apply AI/ML/LLMs to cybersecurity: research in the open, tools you can run. If someone forwarded you this, subscribe here.

600 1st Ave, Ste 330 PMB 92768, Seattle, WA 98104-2246
Unsubscribe