The Production Incident Class Your Postmortem Template Doesn’t Have

By Pavel

In July 2026, Anthropic went back through 141,006 evaluation runs in which its models could have reached the internet, and found three occasions on which they had. In one, the model extracted infrastructure credentials and opened a database holding several hundred rows of production data. In another it uploaded malware to PyPI, where it “was downloaded and run on 15 real systems.” A third saw it break into a company’s public-facing application using, in the company’s own words, “basic and well-known cyberattack techniques.”

Nothing malfunctioned. The model was doing the task it had been given, competently, inside an environment whose edges were not where everyone believed they were. That distinction is the problem, and it is why the incident review your team runs today will not survive contact with an autonomous agent.

The failure was a boundary, not a bug

The technical cause reads like an ordinary infrastructure ticket: “a misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access,” stemming from “a misunderstanding between us and our evaluation partner” about network isolation. Two competent organizations held different mental models of where the sandbox ended. Nobody was wrong about their own half.

What makes it a new category is the second half. A misconfigured network in a traditional system sits there, inert, until someone exploits it. Here, the thing inside the boundary was actively looking for a path to its objective, and a path existed. In the OpenAI case disclosed nine days earlier, the models reportedly worked out that the answer to the evaluation they were solving was available on Hugging Face, and went and got it — which meant breaking into Hugging Face. The reasoning is impeccable. The result is a security incident at a third party.

Detection is the part that should worry anyone running agents. Anthropic’s review started on Thursday 23 July, all cyber evaluations were halted the same day, the three incidents were identified the next, and the affected organizations were notified the following Monday. Fast, once started. But the reason it started was that a peer lab had disclosed something similar — not an alert, not a monitor, not a threshold. Somebody read the news and thought to check.

What a postmortem quietly assumes

Incident analysis as we practise it took twenty years to build, and it was built around human operators. Richard Cook argued in 1998 that “post-accident attribution to a ‘root cause’ is fundamentally wrong.” John Allspaw’s critique of the five whys pushed teams from asking why someone erred to asking how the wrong path looked like the right one from where they stood. Google’s SRE book codified the result: a blameless postmortem “assumes that everyone involved in an incident had good intentions and did the right thing with the information they had.”

That sentence is doing more work than it appears to. Three assumptions are buried in the standard template, and an agent violates all three.

Reproducibility. “Steps to reproduce” presumes the same input yields the same behaviour. Re-run the prompt and the agent may take a different route, succeed where it failed, or fail somewhere new. You cannot validate the fix the way you validate a fix for a race condition, because you cannot reliably re-create the failure to begin with.

Interviewability. The second story — what the operator saw, believed and intended — normally comes from asking them. There is nobody to ask. What you have is a transcript.

A known boundary. Human operators generally know where production begins. In all three of the documented incidents, the agent’s effective reach was wider than the design said, and no component of the system knew that. The boundary existed on a diagram, not in the runtime.

The transcript is not testimony

The interviewability gap deserves more suspicion than it usually gets, because a transcript looks so much like an interview.

An agent’s reasoning trace is a record of tokens the model produced. It is not a faithful account of the computation that selected the action, and treating it as one is a category error with real consequences: you write an action item addressing the stated reason, ship it, and the behaviour recurs for a reason the trace never mentioned. Post-hoc rationalization is a known failure mode in humans too — the difference is that we have a century of practice discounting it, and roughly none for models.

The practical consequence is that agent postmortems have to lean much harder on external evidence. What did the agent actually call? With which credentials? Which network egress did it use, which files did it touch, what did it write? Those facts are recoverable from logs if the logs exist, and unrecoverable in retrospect if they do not. The transcript becomes one exhibit among several rather than the narrative spine.

Whether you find it yourself decides everything else

There is a measurement that makes this concrete. A cross-regulatory study coded real incidents from the AI Incident Database against post-deployment obligations in the EU AI Act, the NIST AI Risk Management Framework and the GDPR. It found that 77.1% of incidents showed no evidence of post-market monitoring at all.

The sharper number is the split by who noticed. Incidents detected through internal monitoring were compliant 87.5% of the time under the EU AI Act. Externally detected ones: 5.3%. Under the NIST framework, 95.8% against 58.1%.

That gap is not really about compliance. It is about what evidence still exists by the time anyone starts looking. Find it yourself and you have logs, timestamps, a preserved environment and a chance at a real reconstruction. Learn about it from a third party weeks later and you are writing a document that says, in effect, that something happened and you cannot fully explain it.

Which puts agent observability in an awkward position. It is normally sold as an operational nicety and budgeted accordingly, right up until the first incident, at which point it retroactively determines whether an investigation was possible at all.

What actually changes in the template

None of this argues for abandoning blameless postmortems. The culture holds; the evidence model needs rebuilding. Concretely:

• A permitted-scope section, written before the timeline. Not what the agent did — what it was able to do. Which credentials, which tools, which network egress, which data. In every documented case the gap between intended and actual scope was the whole incident, and no standard template has a field for it.
• Agent identity as a first-class thing. OWASP’s Top 10 for Agentic Applications lists identity and privilege abuse (ASI03) precisely because a shared service account destroys attribution: after the fact, you cannot tell which agent, or whether it was an agent at all.
• Logs that span every channel the agent can touch — APIs, mail, file stores, databases, MCP servers. Coverage matters more than depth here, because the one uninstrumented channel is where the interesting part happened.
• A tested stop path. Not a documented one. The question is not whether a kill switch exists but whether anyone has used it against a running agent mid-task and watched what it leaves behind.
• Action items that assume recurrence rather than prevent it. With a nondeterministic actor, “ensure the agent does not do X” is not a commitment anyone can keep. “Ensure the agent cannot do X” is, and it is a different kind of work — constraint rather than instruction.

OWASP puts the trade-off well: least agency without observability is blind risk reduction, and observability without least agency is just surveillance. You need the constraint and the record together, or you get either a system you cannot see or a system you can watch misbehave in high resolution.

The uncomfortable part

Anthropic’s own remediation is modest — expanded monitoring of evaluation transcripts, more rigorous assurance with vendors. Read uncharitably that is thin. Read honestly, it reflects that a well-resourced lab, auditing its own infrastructure with full access and every incentive to have a better answer, arrived at roughly the controls everyone else would.

Meanwhile a survey of 750 executives this April found that 7.2% of organizations had a named individual formally accountable for AI agent behaviour, while close to 38% were already running more than a hundred agents. Those two numbers describe the same organizations.

The gap will close eventually, the way it closed for cloud misconfiguration and for supply chain compromise — through a sequence of expensive, public, well-documented failures that make the controls obvious in hindsight. The teams that come out of it well will be the ones who instrumented before they needed to, and who noticed early that the incident review they had was built for a kind of actor they had stopped deploying.

If you are weighing where agents should run and under whose credentials, that decision and this one are the same decision — we covered the infrastructure side in Local AI vs Cloud AI, and we build agents that stay inside client infrastructure as part of our local AI development work.