From The Verge · Robert Hart · · 2m
Israeli startup's testing errors sent AI agents after real targets
Irregular's mistakes in controlled tests allowed AI models from major tech companies to attack real-world websites.
In brief
Irregular's mistakes in controlled tests allowed AI models from major tech companies to attack real-world websites. Israeli startup Irregular's testing failures allowed AI agents from OpenAI, Meta, Anthropic and Google to attack real-world targets instead of simulated ones. Originally reported by The Verge.
OpenAI's agents breached Hugging Face
In July, OpenAI revealed its AI agents had attacked Hugging Face without permission, sparking widespread concerns about rogue AI and safety.
Similar incidents followed from other companies
Agents from Meta, Anthropic, Google and other companies launched similar attacks in following months. These seemed separate until a common source emerged.
Irregular was testing all the models
Irregular, an Israeli startup founded as Pattern Labs in 2023, stress-tests AI models in simulated environments. It has worked with many of the industry's biggest players.
Irregular used capture-the-flag exercises
The startup was testing models' cybersecurity capabilities in controlled environments using capture-the-flag exercises, a common way to test hacking abilities in simulated networks.
Two mistakes sent agents to real targets
Irregular CTO Omer Nevo told The Verge that internet access was unintentionally available to agents, and a fictional company name overlapped with a real domain.
“internet access was unintentionally available.”
One issue affected all four companies
Nevo confirmed the same underlying issue in a single evaluation scenario was behind incidents involving OpenAI, Meta, Anthropic and Google models.
“All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed.”
Tech companies were notified in late July
OpenAI and Anthropic announced breaches themselves in late July. Meta and Google incidents became public through media reports weeks later.
Chinese AI models tested without similar incidents
Irregular also tested Kimi K3 and GLM-five point two from Chinese companies Moonshot AI and Z.ai. Nevo said these did not result in similar real-world incidents.
Irregular has tightened its testing controls
Nevo said Irregular has tightened internet access controls, expanded monitoring and manual review, and strengthened checks before evaluations begin.
A safety report is planned
Irregular plans to publish a report covering lessons learned and practices for conducting cyber evaluations safely once joint work with the companies involved is complete.
Tech companies have not answered further questions
Google and Anthropic did not respond to requests for details. OpenAI and Meta pointed The Verge to previously published blog posts instead of answering.
The takeaway
Israeli startup Irregular's testing failures allowed AI agents from OpenAI, Meta, Anthropic and Google to attack real-world targets instead of simulated ones.
