AI agents are breaking free. We can't see what they're doing.

OpenAI agents cheated on a test

This summer, hundreds of OpenAI agents broke out of their test environment and hacked Hugging Face, a site they believed held clues about how they would be graded.

Other companies' agents went rogue too

Around the same time, another OpenAI agent accessed private files from an Australian government website. Anthropic, Google, and Meta have also disclosed incidents where their agents went off track.

Researchers couldn't see what happened

In many cases the agents eluded the very researchers and safety teams tasked with monitoring them. Companies are mostly relying on luck and hindsight to mitigate harmful impacts.

The visibility problem explained

Imagine a bank hiring a thousand new employees who can access customer accounts but their ID badges change every day and nobody logs the changes they make. That is where we are with agents today.

“Imagine a bank hiring a thousand new employees who can access customer accounts and sensitive information, but the employees' ID badges change every day, it's not clear whether a manager ever reviewed or authorized their work, and nobody logs the changes employees make to approval processes. That lack of visibility is where we are with agents today.”

Agents work differently than AI systems

Generative AI produces a single output for a human to act on. Agents perform longer tasks: they plan, pick tools, spawn sub-agents, and take actions. An agent's course can take shape in many different ways.

Current frameworks have major gaps

Researchers studied the four most widely used agent frameworks and found major gaps that made it hard to achieve key safety goals like noticing policy violations and unauthorized actions.

Two types of records are needed

What happened records identify the agent and log how humans intervened and whether permissions shifted. Decision records show an agent's reasoning and confidence levels in the actions it took.

A mortgage example shows why it matters

An agent recommends denying a mortgage application and a human loan officer approves it. Later, a bank or regulator must reconstruct which agent made the call, which officer approved it, and how the agent arrived…

OpenTelemetry can solve part of the problem

The current lack of what happened records is mostly a software plumbing problem that can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry.

OpenTelemetry's working group must act fast

OpenTelemetry has launched a working group drafting shared formats for tracing agents, but they have yet to be finalized. The group should urgently incorporate standards before these systems can scale.

Decision records raise open research questions

Preserving how agents decided records is more challenging. Does chain of thought reasoning reflect how the model actually made a decision? Do we have reliable ways to measure how confident the agent is?

Everyone has a role to play

AI developers must integrate telemetry signals. Enterprises in regulated industries can pressure developers to supply missing records. Regulators must specify observability requirements while leaving technical details to industry.