# AI physics benchmarks are broken, not the models

> Expert review reveals that frontier AI models score far higher on physics than reported, exposing flawed evaluation methods.
- Title: AI physics benchmarks are broken, not the models · arXiv
- Summary: Expert review reveals that frontier AI models score far higher on physics than reported, exposing flawed evaluation methods. Physics benchmarks dramatically…
- Keywords: frontier models, physics benchmarks, evaluation methodology, gpt-5.6-sol, benchmark auditing, expert review, ai reasoning, quantitative problem-solving, hle-physics, cmt-benchmark, technology, startups
- Source: arXiv — https://arxiv.org/abs/2609.13009
- Read time: 1 min
- Topics: frontier models, physics benchmarks, evaluation methodology, gpt-5.6-sol, benchmark auditing, expert review, ai reasoning, quantitative problem-solving, hle-physics, cmt-benchmark, technology, startups
## Physics benchmarks show AI struggling
Leading physics benchmarks report that frontier language models score poorly on advanced physics problems, suggesting they lack scientific reasoning and quantitative problem-solving abilities.
## But experts see something different
Domain experts using these models in their actual work report experiences that don't match the low benchmark scores, creating a disconnect between reported performance and real-world capability.
## Researchers audited the benchmarks
Physics faculty and graduate researchers reviewed problems, solutions, and model responses across six major benchmarks to identify whether low scores reflected actual model errors or problems with the evaluation itself.
## Most errors weren't from the AI
Most cases marked as incorrect turned out to have grading errors, wrong reference solutions, or ambiguous questions. The benchmarks themselves were flawed, not the models' physics reasoning.

> Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning.
## Corrected scores are dramatically higher
After experts fixed flawed questions and reference solutions, GPT-5.6-Sol's score on HLE-Physics jumped from 47.3% to 78.7%, and on CMT-Benchmark from 61.0% to 87.2%.
## Some benchmarks nearly saturated
On the 54 retained CritPt challenges, GPT-5.6-Sol achieved 94.4% pass rate after corrections, showing frontier models have nearly maxed out these closed-ended physics tasks.
## Benchmarks need to get harder
The findings reveal that current physics benchmarks substantially understate frontier models' actual abilities and highlight the urgent need for more demanding, expert-validated evaluations.

> Current benchmarks substantially understate frontier models' ability to solve well-posed physics problems.
## Key takeaway

Physics benchmarks dramatically underestimate frontier models' abilities due to grading errors and poorly specified questions, not model limitations.