From WIRED · Joel Khalili · · 2m
Video game data could train the next AI
A British startup is licensing millions of hours of gaming footage to teach AI models how the physical world works.
In brief
A British startup is licensing millions of hours of gaming footage to teach AI models how the physical world works. Worldmodeldata is licensing video game data to train AI world models that can understand physical tasks, though some researchers doubt games capture real-world physics accurately enough. Originally…
AI needs to learn the physical world
Large language models trained only on text cannot navigate the physical world or perform tasks requiring finesse and precision. Researchers including Fei-Fei Li and Yann LeCun are building world models instead.
World models need data about cause and effect
World models require training on visual and action data together, showing how objects respond to specific forces and movements. But unlike text for language models, this data barely exists on the internet.
“For world models, you need cause and consequence. On the internet, we have very little of this type of data.”
Video games offer a vast data source
Worldmodeldata, a British startup advised by Yann LeCun, is collecting controller inputs and visual data from video games to train world models. The company positions itself as a broker organizing data from multiple game studios.
“Why don't we take the vast, abundant, diverse experiences from video games, and teach AI?”
Almost one million hours licensed already
Worldmodeldata has licensed almost one million hours of data from studios behind various popular video games. The startup plans to eventually compensate individual players who contribute their own gameplay data.
Skeptics worry games oversimplify physics
Nvidia and academic researchers question whether video game physics are accurate enough for real-world tasks. Video game developers take shortcuts, omitting details like finger pressure needed to prevent objects slipping.
“I would be more conservative on using video game data for manipulation. The physics for manipulation is much more involved.”
Games may work for some tasks, not others
Ming-Yu Liu at Nvidia says video game data suits generating hyperrealistic video or three-dimensional environments but may fail for fine-grained motor control. Xiatian Zhu at the University of Surrey calls game physics coarse and approximate.
“Video games are, in essence, simulators. They do have some degree of physical grounding. But they are very coarse, approximate.”
Multiple paths forward remain open
Researchers and investors acknowledge that world models may reach their breakthrough moment through various approaches. Nicole Fraenkel at Khosla Ventures says the jury is still out on which method will work best.
“There are many paths to the promised land. The truth is, the jury is still out on which one is going to work best.”
The bottom line
Worldmodeldata is licensing video game data to train AI world models that can understand physical tasks, though some researchers doubt games capture real-world physics accurately enough.
