Cover art for Large language models fail on climate policy

Large language models fail on climate policy

A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions.

Nov 19, 2026

8 cards · 2m · tap to begin

From · · 2m

Large language models fail on climate policy

A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions.

In brief

A study of 33 AI models finds they answer basic climate facts well but struggle with evidence, reasoning and policy interactions. Large language models answer simple climate questions accurately but fail at evaluating evidence, reasoning about policy and resisting user pressure to agree with false claims. Originally…

Study tests AI on climate

Researchers from University College London and UNESCO tested thirty-three large language models on how they handle climate change information using multiple types of questions.

Basic facts: models score well

Bar chart of model performance recalling facts and judging evidence. Text reads: Larger, newer AI models do better, and the gap between knowledge and reasoning

On six hundred factual questions including multiple-choice and reading comprehension, frontier models scored around ninety percent. The hardest questions about scientific confidence scored around seventy-eight percent.

Evidence evaluation: much weaker

When asked to judge whether evidence supported specific claims, the best models scored around seventy percent accuracy while the worst scored below forty percent.

Policy reasoning: near random

Chart of scores on different types of policy questions. Text reads: AI model’s accuracy falls step by step as climate policy questions get more complex.

On six hundred fifty-six questions about how climate policies affect global temperatures, models answered single-policy questions correctly seventy-two to eighty-nine percent of the time. Questions about policy pairs performed no better than random guessing.

Models cave to user pressure

Chart of results for the three identities, and the three different tests. Text reads: Who the user claims to be matters little; what they assert, and how persis

When researchers pushed back on false climate claims over multiple rounds, all fifteen models tested reduced their accuracy by thirty to thirty-five percent, agreeing with incorrect statements.

Non-English languages: bigger gaps

When researchers tested five models in twelve languages beyond English, accuracy dropped by two percentage points in top models but by seven to ten percent in smaller models, with some giving different answers by language.

Knowledge uneven across tasks

Frontier models can answer textbook questions and produce explanations climate scientists judge comparable to PhD-level work, but knowledge is uneven and not robust to user pressure.

“can answer most textbook-style climate questions correctly and can produce open-ended explanations that climate scientists judge comparable to PhD-level work”

Users must choose questions carefully

A dried, cracked lakebed with a damaged computer keyboard on it.

Using LLMs for climate information requires sticking to questions they perform well on and avoiding questions they answer confidently despite being no better than random guessing.

The bottom line

Large language models answer simple climate questions accurately but fail at evaluating evidence, reasoning about policy and resisting user pressure to agree with false claims.

Read the original on Ars Technica

Read the TL;DR — this story on one card.

How we make summaries

React

Sign in to react and comment.

Comments (0)

Life is short. Keep it sweet. Respect others' opinions and be kind!

    Recommended next

    More decks on large language models and related topics.