Large language models (LLMs) like ChatGPT are often accused of bias. But what does “bias” even mean in the context of an AI? New research from KAIST explores the tricky problem of defining and measuring bias in LLMs, revealing that our current methods might be painting an incomplete picture. The researchers argue that judging bias solely on whether an LLM treats all demographic groups equally isn't enough. They propose a “fact-based” approach, comparing LLM outputs to real-world statistics. For instance, if an LLM suggests “nurse” more often for female personas, that might seem biased at first glance. However, if the proportion of female nurses generated by the LLM roughly matches the actual proportion of nurses who are women, is the model truly biased or simply reflecting reality? A human survey conducted as part of the study supports this statistically aligned view, finding that people generally prefer LLM-generated responses that correspond to real-world demographics. The research also reveals that different models exhibit varying degrees of bias depending on the metrics used. Some prioritize balance, ensuring an even distribution across demographics, while others seemingly align their responses more with real-world statistics. The study shows how the method of “fine-tuning” LLMs to be more helpful and harmless (using techniques like reinforcement learning with human feedback) can sometimes actually exacerbate biases or even create anti-stereotypical behaviors. For example, overly aggressive debiasing might cause a model to drastically underrepresent a demographic group in a particular occupation, contradicting real-world statistics. This underscores the importance of assessing bias from multiple perspectives. While aligning LLM outputs with real-world data offers a promising approach, the researchers emphasize the need for more nuanced analysis, considering the potential for bias within the statistical data itself. The road to truly unbiased AI is complex, and this research reminds us that simply striving for equal representation across all groups may not be the answer. A more nuanced, statistically-informed approach is crucial for building AI systems that are both fair and reflective of the complex world we live in.