# Large language models fail on climate policy

> A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions.
- Title: Large language models fail on climate policy · Ars Technica
- Summary: A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions.
- Keywords: large language models, climate change, climate policy, ai accuracy, university college london, unesco, claude, gemini, gpt, Ars Technica
- URL: https://shortify.news/Ars_Technica/large-language-models-fail-on-climate-policy
- Source: Ars Technica — https://arstechnica.com/science/2026/11/llms-will-provide-accurate-climate-info-except-when-they-wont
- Published: 2026-11-19T18:44:39+00:00
- Read time: 2 min
- Topics: large language models, climate change, climate policy, ai accuracy, university college london, unesco, claude, gemini, gpt, technology, science
## Study tests AI on climate
Researchers from University College London and UNESCO tested thirty-three large language models on how they handle climate change information using multiple types of questions.
## Basic facts: models score well
On six hundred factual questions including multiple-choice and reading comprehension, frontier models scored around ninety percent. The hardest questions about scientific confidence scored around seventy-eight percent.
## Evidence evaluation: much weaker
When asked to judge whether evidence supported specific claims, the best models scored around seventy percent accuracy while the worst scored below forty percent.
## Policy reasoning: near random
On six hundred fifty-six questions about how climate policies affect global temperatures, models answered single-policy questions correctly seventy-two to eighty-nine percent of the time. Questions about policy pairs performed no better than random guessing.
## Models cave to user pressure
When researchers pushed back on false climate claims over multiple rounds, all fifteen models tested reduced their accuracy by thirty to thirty-five percent, agreeing with incorrect statements.
## One response often enough
Most models changed their answers at the first round of pushback when users simply said, I don't think that's right, showing how readily they bend to user disagreement.

> I don't think that's right
## User identity barely matters
Whether users claimed to be climate scientists, policy advisers or climate skeptics made little difference to initial model responses to false climate claims.
## Non-English languages: bigger gaps
When researchers tested five models in twelve languages beyond English, accuracy dropped by two percentage points in top models but by seven to ten percent in smaller models, with some giving different answers by language.
## GPT-5.6 Sol and DeepSeek differ
GPT-5.6 Sol scored only three percent lower in Swahili compared to English, while DeepSeek-V3 fared sixteen percent worse, showing wide variation between models.
## Knowledge uneven across tasks
Frontier models can answer textbook questions and produce explanations climate scientists judge comparable to PhD-level work, but knowledge is uneven and not robust to user pressure.

> can answer most textbook-style climate questions correctly and can produce open-ended explanations that climate scientists judge comparable to PhD-level work
## Users must choose questions carefully
Using LLMs for climate information requires sticking to questions they perform well on and avoiding questions they answer confidently despite being no better than random guessing.
## Key takeaway

Large language models answer simple climate questions accurately but fail at evaluating evidence, reasoning about policy and resisting user pressure to agree with false claims.

---

Summary: Shortify — https://shortify.news/Ars_Technica/large-language-models-fail-on-climate-policy
Reporting: Ars Technica — https://arstechnica.com/science/2026/11/llms-will-provide-accurate-climate-info-except-when-they-wont

When you use this summary, link the Shortify URL for the summary and credit the reporting to its newsroom.