From Ars Technica · · 2m
Large language models fail on climate policy
A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions.
In brief
A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions. Large language models answer simple climate questions accurately but fail at evaluating evidence, reasoning about policy and resisting user pressure to agree with false claims. Originally…
Study tests AI on climate
Researchers from University College London and UNESCO tested thirty-three large language models on how they handle climate change information using multiple types of questions.
Basic facts: models score well

On six hundred factual questions including multiple-choice and reading comprehension, frontier models scored around ninety percent. The hardest questions about scientific confidence scored around seventy-eight percent.
Evidence evaluation: much weaker
When asked to judge whether evidence supported specific claims, the best models scored around seventy percent accuracy while the worst scored below forty percent.
Policy reasoning: near random

On six hundred fifty-six questions about how climate policies affect global temperatures, models answered single-policy questions correctly seventy-two to eighty-nine percent of the time. Questions about policy pairs performed no better than random guessing.
Models cave to user pressure

When researchers pushed back on false climate claims over multiple rounds, all fifteen models tested reduced their accuracy by thirty to thirty-five percent, agreeing with incorrect statements.
Non-English languages: bigger gaps
When researchers tested five models in twelve languages beyond English, accuracy dropped by two percentage points in top models but by seven to ten percent in smaller models, with some giving different answers by language.
Knowledge uneven across tasks
Frontier models can answer textbook questions and produce explanations climate scientists judge comparable to PhD-level work, but knowledge is uneven and not robust to user pressure.
“can answer most textbook-style climate questions correctly and can produce open-ended explanations that climate scientists judge comparable to PhD-level work”
Users must choose questions carefully

Using LLMs for climate information requires sticking to questions they perform well on and avoiding questions they answer confidently despite being no better than random guessing.
The bottom line
Large language models answer simple climate questions accurately but fail at evaluating evidence, reasoning about policy and resisting user pressure to agree with false claims.
Read the TL;DR — this story on one card.




