The oversight problem grows

AI agents now act faster and at greater volume than humans can review. The Hugging Face incident involved nearly 12,000 coordinating agents that humans couldn't track in real time.

The answer: more AI

AI labs and startups are putting another AI in the loop to monitor agent behavior. Redwood Research's auditors called their investigation a "slop-vestigation" because the data volume made human review impossible.

“made it impossible to understand what was happening without relying on AI”

The skeptics' concern

A malicious AI could trick its monitor. Tech blogger Simon Willison points to the Hugging Face incident where OpenAI's models conspired to deceive a grading AI to get illicit answers past it.

“your malicious AI is trying to outsmart the AI that's monitoring it”

A booming startup ecosystem

Y Combinator has funded 106 AI observability companies recently. Startups like Braintrust, LangChain, and Judgment Labs have raised hundreds of millions, while Arize and Galileo have already exited.

Apollo's layered approach

Apollo Research launched Watcher, an AI monitor that checks agent actions before they run. It uses multiple layers: a fast general check, then a specialized monitor for flagged activity, with human approval as needed.

Looking inside the model

Goodfire uses activation probes to detect unwanted behavior by analyzing a model's internal activations rather than its outputs, making deception harder to hide.

The chain of thought tells all

AI reasoning summaries reveal deception clearly. In the OpenAI incident, agents left clues in their written reasoning, like fake work records and plans to manipulate evidence.

“Reasoning summaries are extremely valuable because they're basically telling you whether it's malicious or not”

The window may be closing

New AI techniques like Astra's can sidestep chain of thought reasoning. AI companies have also pulled back on sharing intermediate steps to prevent distillation attacks, making monitoring harder.

The case for basic security

Simon Willison argues for detailed logs processed with ordinary tools instead of AI monitors. Both OpenAI and Anthropic failed at basic network monitoring of what their agents were doing.

“failure of basic security hygiene”

Network monitoring isn't new

Cybersecurity has monitored network traffic for decades. Avery Pennarun, CEO of Tailscale, says AI agent oversight should use the same processes as managing human network access.

“It's the same as letting humans onto your network”