From Business Insider · Stephen Council · · 1m 20s
AI giants call for internal watchdogs to police their own risks
OpenAI and Anthropic want third-party evaluators embedded in their companies, and AI pioneers are setting strict conditions for how that should work.
In brief
OpenAI and Anthropic want third-party evaluators embedded in their companies, and AI pioneers are setting strict conditions for how that should work. Major AI labs are opening their doors to outside oversight, but safety experts say evaluators need real independence, transparency rights, and protection from…
AI labs want internal watchdogs
OpenAI and Anthropic committed this week to letting third-party evaluators work inside their companies with employee-level access to assess AI risks.
Safety experts set strict conditions
The AI Evaluator Forum, backed by Geoffrey Hinton and other pioneers, released a letter Friday outlining minimum standards for how these embedded evaluators should operate.
Evaluators need real independence
The letter demands that evaluators have scientific objectivity, unfiltered access to company boards, the right to publish findings publicly, and protection from retaliation.
“To be credible, embedded third-party evaluators must have scientific objectivity, transparency, independence, and robust protections against interference.”
Access to systems and data required
Evaluators should get the same access to relevant systems, data, tools, and physical spaces as the companies' own safety assessors to do their jobs properly.
Diverse expertise across risk types
Labs should hire evaluators with different viewpoints and expertise, including specialists in biological risks and loss of control scenarios.
Tech leaders back the approach
Microsoft CEO Satya Nadella and other industry figures have endorsed the embedded evaluators idea as a way to build trust in AI development.
The one thing
Major AI labs are opening their doors to outside oversight, but safety experts say evaluators need real independence, transparency rights, and protection from retaliation to be credible.
