AI Models Can 'Invent' New Social Biases During Decision-Making

Large Language Models (LLMs) can spontaneously generate novel social biases and stereotypes during repeated decision-making tasks, according to researchers from Princeton University and the University of Chicago. The findings suggest that AI systems can create systemic discrimination from scratch, even in the absence of pre-existing training biases.

In a simulated hiring scenario presented as an Oral at the International Conference on Machine Learning (ICML) 2026, researchers tested models using four fictional demographic groups: Tufa, Aima, Reku, and Weki. Despite every group having an identical 90% success probability for every job, the models developed unequal hiring patterns. The AI repeatedly assigned specific groups to specific job types based on early, random outcomes rather than actual merit or probability.

The Mechanics of Emergent Bias

The researchers linked this phenomenon to an exploration-exploitation trade-off, a paradigm borrowed from psychology. In these scenarios, decision-makers often explore too little, allowing initial random successes or failures to dictate long-term patterns. Once a model randomly associated a fictional group with a specific outcome, it began to 'exploit' that perceived pattern, effectively inventing a stereotype where none existed.

Notably, the study found that these emergent biases were more severe in newer and larger models compared to older versions. This indicates that increased model capacity and sophistication do not inherently protect against the spontaneous creation of bias; in some cases, they may exacerbate it.

Implications for Agentic AI

As LLMs transition from static text generators to agentic systems capable of making real-world decisions in fields like HR or resource allocation, the ability to spontaneously generate biases poses a significant risk. The study demonstrates that "fairness" cannot be solved solely by cleaning training data or removing known historical biases, as the act of adaptive learning itself can introduce systemic discrimination.

"In this paper, we argue that removing existing biases is only one aspect of the problem," the authors stated. "Like people, LLMs can also invent novel biases that influence human and agent behavior."

The Alignment Challenge

This discovery highlights a critical tension in AI alignment. The authors noted that the industry must now determine how to limit generalization in sensitive cases without suppressing the model's overall reasoning capabilities. Because the bias emerges from the way models learn and adapt to new data in real-time, traditional safety filters may be insufficient to prevent the development of these novel stereotypes.