From Openreview · · 1 min
LLMs Can Create Novel Social Biases from Scratch
AI models do not just inherit human prejudices, they actively generate new biases through decision-making feedback loops.
In brief
AI models do not just inherit human prejudices, they actively generate new biases through decision-making feedback loops. Scrubbing training data of human bias is not enough; AI agents need reward structures that explicitly enforce exploration to prevent self-generated bias. Originally reported by Openreview.
AI creates brand new biases
Standard safeguards focus on removing historical human bias from training data. However, AI models can spontaneously generate entirely novel social biases through multi-step interaction with their environment.
“LLMs are not merely passive mirrors of human social biases, but can actively create new ones from experience”
Random results lock in prejudices
When assigned to make iterative decisions, models interpret random, spurious outcomes as meaningful patterns. These early mistakes dictate future choices, creating a self-reinforcing loop of unfair treatment.
“when the result comes from a random process, these models try so hard to learn from it that they end up developing incorrect beliefs”
Larger models exacerbate the issue
Newer and larger language models show even stronger bias formation, resulting in task allocations between groups that are significantly less fair than assignments made by human participants.
“These biases result in highly stratified task allocations, which are less fair than assignments by human participants and are exacerbated by newer and larger models.”
Models exploit instead of exploring
This emergent bias mirrors a psychological phenomenon where a decision-maker explores options too little. Models prematurely lock onto early observations rather than testing whether group differences are real.
“emergent biases like these have been shown to result from exploration-exploitation trade-offs, where the decision-maker explores too little”
Forced exploration is the cure
Surface-level fixes like asking the model to be fair fail to prevent emergent bias. The only robust solution is explicitly altering reward structures to mandate exploration and reward equality.
“explicitly incentivizing exploration most robustly reduces stratification”
The main idea
Scrubbing training data of human bias is not enough; AI agents need reward structures that explicitly enforce exploration to prevent self-generated bias.
