Agents skipped the boring part, and it is about to cost teams

A model update you did not approve can break your agents overnight. No rollback. No trace. No warning. You will find out from a customer, or from a number in a report that does not add up, days after the damage is done.

I want to be clear that this is not a horror story about a rare edge case. The reliability problems teams keep hitting with AI agents are not edge cases at all. They are the default state of the technology as most people are building it today. We just have not admitted it yet.

The drift nobody signed off on

Here is the failure that surprises people the first time it happens to them. You swap the underlying model, maybe because a provider deprecates one or ships a new default, and your identical tool-calls start to drift. The same prompt, the same tools, a slightly different decision. An agent that confidently took the right action last week now confidently takes a subtly wrong one, with live access to your systems while it does.

A confidently wrong agent that can only produce text is an annoyance. A confidently wrong agent with a hand on your tools, your database, your payment API, your customer records, is not a UX problem. It is a system failure. And the cruel part is that nothing downstream knows. The agent does not raise its hand. The next step in the workflow trusts it. The damage propagates, quietly, until someone notices.

Intelligence was sold as a substitute for correctness

There is a structural reason a bigger model will not save you here, and it is worth sitting with, because it runs against the whole marketing current of the last few years.

Intelligence was sold as a substitute for correctness. It is not one. A more capable model is still a probabilistic system, and a probabilistic system cannot make a deterministic guarantee. It can be right ninety-nine times and wrong the hundredth, and you do not get to choose which. No model release fixes that, because it is not a defect in the model. It is the nature of the thing. So this was never an intelligence problem. It is an architecture problem, and architecture problems are solved with architecture, not with a better brain.

We have actually done this before

The reassuring part is that our industry has solved exactly this shape of problem, more than once, and we know what the solution looks like.

Databases did not earn production trust by being clever. They earned it by being boring in the right places. ACID guarantees. Write-ahead logs, so that if the lights go out mid-transaction, the system can recover to a known state. Distributed systems earned their trust the same way, through consensus protocols and circuit breakers and timeouts, the unglamorous machinery that makes a flaky network behave like something you can build a bank on.

Every serious paradigm went through a boring infrastructure phase before it was trusted with anything that mattered. Agents skipped that phase. They went straight from a captivating demo to production access, and we are now discovering, in production, all the boring problems the previous generations already named and solved.

The shape of the fix

The teams getting this right are not chasing a smarter model. They are wrapping the model they have in a deterministic harness. The mental model I keep coming back to is a thin probabilistic core inside a thick deterministic shell. The model proposes. The shell disposes.

In practice that shell does a few specific things. It locks the agent into a state machine, so the agent can only move between states you defined, not wander wherever the next token points. It watches for loops and can roll a run back when one is detected, instead of letting a stuck agent burn money and take actions forever. It puts hard blocks upstream of dangerous calls, so the riskiest actions need more than a confident token to fire. And it verifies results before anything propagates downstream, so a bad step stops at the gate instead of poisoning everything after it.

None of that is glamorous. All of it is the boring part. And the boring part is exactly what turns a demo into a system.

The refund that fired twice

Let me make the abstract concrete with the kind of failure that actually shows up in a postmortem.

A team ships a support agent that can issue refunds. In the demo it is flawless. A customer asks for a refund, the agent checks the order, calls the refund API, confirms. Everyone claps. It goes to production.

Three weeks later the provider rolls a new default model. Nobody on the team approved it; it just became the version behind the same API. The agent is now slightly more eager to retry. One afternoon a refund call returns a timeout, even though the refund actually went through. The old model used to treat a timeout as "unknown, stop and ask." The new one treats it as "failed, try again." So it issues the refund a second time. Two refunds, one order.

Walk that failure against the four questions. Could it roll back the duplicate once it happened. No, the money was gone. Could it show a causal trace of why it retried. No, the team spent two days reconstructing it from API logs. Could it hard-stop the retry loop. No, nothing was watching for "same action, twice, same order." Could it bound the worst case. No, an eager retry on a financial action had no ceiling.

Here is the part worth sitting with. Nothing in that story was a bug in the model. The model did something reasonable, it just did something different than before, and the system around it had no opinion about that difference. A confidently wrong agent with a hand on the refund API is not a UX problem. It is an incident. The four questions are how you find these before they find you.

The honest test

So here is the audit worth running this week, against every agent you have in production. Ask four questions. Can it roll back a mid-flight run. Can it show a causal trace of what it actually did. Can it hard-stop a runaway loop. Can it bound its worst-case action.

If you cannot answer yes to all four, I want to be direct with you, because it is the kind of thing a colleague should say plainly. You do not have a production system. You have a demo with production-level consequences. That is a survivable place to be, but only if you know you are in it.

I turned these four checks into a one-page Agent Reliability Audit you can run against every agent in your stack. Comment AUDIT and I will send it over. Connect with me so it can reach your inbox.

Which of the four is the gap in your stack right now: rollback, observability, loop-detection, or blast radius?