From MIT Technology Review · · 3m
AI still can't do the research that builds better AI
Princeton researchers gave AI agents six days, $3,000, and a GPU budget to write a conference paper. The authors rejected both.
In brief
Princeton researchers gave AI agents six days, $3,000, and a GPU budget to write a conference paper. The authors rejected both. Today's agents are strong engineers and weak researchers, and the creative leaps that produced things like transformers are exactly what they can't yet make. Originally reported by MIT…
The industry's boldest promise
AI labs say models will soon improve themselves with almost no human oversight. LLMs already write code, generate training data, and tune the chips they run on.
A new study says not yet
Researchers found AI agents can't do open-ended AI research, the free-form kind with no clear answer that needs judgment and taste.
Who ran it
A multi-institution group led by Peter Kirgis and Sayash Kapoor at Princeton tested whether agents could produce work at the level of a top machine-learning conference.
Benchmarks miss the point
Most evaluations check narrow tasks with verifiable answers. Real research means picking hypotheses, deciding what evidence settles a question, and knowing when to start over.
Shadow evaluation
The team's method: hand the agent a research question from a high-quality unpublished paper, so the answer exists but cannot be memorized or googled.
Two unpublished questions
Claude Opus 4.8 on OpenClaw tackled questions from two NeurIPS 2026 submissions: controlling an LLM's personas by editing weights, and detecting when a spreadsheet-data model goes unreliable.
Six days and $3,000
Agents got API credits, a GPU budget, their own virtual machines, and open web access. The papers' real authors then graded the output like conference submissions.
Both papers rejected
The engineering held up. Agents reviewed the literature, ran hundreds of experiments, and compiled results. The research itself did not.
“On the other hand, the agents were unambiguously bad at carrying out the research itself”
They quit on good ideas
Agents formed ambitious hypotheses close to the ones the human authors started with, then abandoned them on thin data. They could make small pivots but never rethink from scratch.
Caveats instead of fixes
Given feedback from subagents or AI reviewers, agents narrowed their claims and added hedges rather than revising the method. They also blew through time and token budgets.
No cheating, at least
The agents didn't reward hack. Subagents sometimes hallucinated results, but the lead orchestrator agent caught them.
Blame the training
Reinforcement learning works on tasks you can score automatically. Kapoor says it's much harder to build training environments when the task itself is open-ended.
“But it's harder to create environments to train these models when the task itself is open-ended”
What the study can't tell you
Only two papers, and the graders knew AI wrote them. The researchers also had wide discretion in the design, so their priors could have leaked in.
The trillion-dollar question
Anthropic's Jack Clark calls the missing creativity a bearish signal on short timelines. Whether self-improvement needs creative leaps, or just faster training and higher scores, is still open.
“That's frankly the trillion-dollar question right now”
The one thing
Today's agents are strong engineers and weak researchers, and the creative leaps that produced things like transformers are exactly what they can't yet make.
