When Agents Break

There’s a rebrand happening in tech right now.

What we used to call “software” is increasingly being called “agents.” They schedule meetings, triage tickets, respond to customers, and even make decisions. The promise is compelling: instead of tools that wait for instructions, we now have systems that act.

But every new abstraction eventually meets reality. And right now, that reality is friction. When agents break, they don’t just fail quietly. They fail publicly, ambiguously, and sometimes expensively.

Press enter or click to view image in full size

The Illusion of Autonomy

Agents are often described as autonomous. In practice, they’re closer to guided systems operating within a loose boundary of rules, prompts, and integrations. That distinction matters.

When traditional software fails, the failure is usually deterministic. A bug can be traced. A system crashes, and logs tell you why. But when an agent fails, the outcome is often harder to explain.

It didn’t “crash.” It made a decision.
And that decision might have been wrong.

This creates a new kind of failure mode: one where the system appears confident, even when it shouldn’t be.

A Familiar Warning Sign

We’ve seen this before.

In 2016, Microsoft Tay was released as an experimental conversational agent on Twitter. Within hours, it began generating offensive and harmful content after learning from user interactions. Microsoft shut it down within a day.

Tay wasn’t a scheduling assistant or a workplace tool, but the lesson still holds: systems that learn and act in open environments can drift in unpredictable ways.

Today’s agents are more advanced, more controlled, and often deployed in enterprise settings. But they still operate on probabilistic reasoning, not certainty. That means unexpected outputs aren’t edge cases — they’re part of the system.

When “Agent” Is Just a New Label

There’s another layer to this conversation that’s less technical and more economic. Some labor economists argue that “agents” are, in many cases, just SaaS with a different interface.

Instead of dashboards, you get chat. Instead of workflows, you get “autonomy.” But underneath, many systems still rely on structured APIs, predefined logic, and human oversight.

The question becomes: are companies actually “hiring agents,” or are they buying more flexible software? The distinction matters because it shapes expectations.

If you think you’re hiring something like a digital employee, you expect judgment, reliability, and accountability. If you’re buying software, you expect constraints, limitations, and the need for configuration.

The Cost of Being Almost Right

One of the most interesting — and dangerous — traits of agents is that they are often almost right.

A scheduling agent might book the correct meeting… but in the wrong time zone. A support agent might answer a question… but miss a critical detail. A sales agent might draft a perfect email… with one incorrect assumption.

These aren’t catastrophic failures. They’re subtle ones. And subtle failures are harder to detect, especially at scale.

In traditional systems, errors tend to be binary. In agent systems, errors exist on a spectrum. That makes monitoring, debugging, and trust much more complicated.

Designing for Failure, Not Just Success

If agents are going to become part of everyday work, the conversation needs to shift.

Not just what can agents do?
But how do they fail?

Good agent design isn’t about eliminating failure entirely. That’s unrealistic. It’s about making failure visible, understandable, and recoverable.

That might look like:

• Clear audit trails of decisions
• Confidence indicators instead of definitive answers
• Easy human override at critical steps
• Guardrails that limit the blast radius of mistakes

In other words, treating agents less like magic and more like systems that require careful design.

Press enter or click to view image in full size

So, What Are We Really Building?

There’s something genuinely powerful about agents. They reduce friction. They collapse interfaces. They make software feel more natural to interact with. And in many cases, they unlock real productivity gains.

But calling them “agents” doesn’t change the fundamental challenge: we’re still building systems that operate in complex, messy environments.

Sometimes they will perform beautifully. Sometimes they will fail in ways we didn’t anticipate.

The companies that succeed in this space won’t be the ones that pretend agents are flawless. They’ll be the ones that understand their limits — and design accordingly. Because the real question isn’t whether agents will break. It’s whether we’re ready when they do.