Physics benchmarks show AI struggling

Leading physics benchmarks report that frontier language models score poorly on advanced physics problems, suggesting they lack scientific reasoning and quantitative problem-solving abilities.

But experts see something different

Domain experts using these models in their actual work report experiences that don't match the low benchmark scores, creating a disconnect between reported performance and real-world capability.

Researchers audited the benchmarks

Physics faculty and graduate researchers reviewed problems, solutions, and model responses across six major benchmarks to identify whether low scores reflected actual model errors or problems with the evaluation itself.

Most errors weren't from the AI

Most cases marked as incorrect turned out to have grading errors, wrong reference solutions, or ambiguous questions. The benchmarks themselves were flawed, not the models' physics reasoning.

“Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning.”

Corrected scores are dramatically higher

After experts fixed flawed questions and reference solutions, GPT-5.6-Sol's score on HLE-Physics jumped from 47.3% to 78.7%, and on CMT-Benchmark from 61.0% to 87.2%.

Some benchmarks nearly saturated

On the 54 retained CritPt challenges, GPT-5.6-Sol achieved 94.4% pass rate after corrections, showing frontier models have nearly maxed out these closed-ended physics tasks.

Benchmarks need to get harder

The findings reveal that current physics benchmarks substantially understate frontier models' actual abilities and highlight the urgent need for more demanding, expert-validated evaluations.

“Current benchmarks substantially understate frontier models' ability to solve well-posed physics problems.”