AI image generators are amazing at creating stunning visuals from text prompts, but how do we ensure they don't sacrifice historical accuracy for the sake of diversity? A new study unveils the "factuality tax" of diversity interventions in text-to-image models. Researchers found that while prompts encouraging diverse representations are effective, they can lead to historically inaccurate depictions, particularly when generating images of real historical figures. Imagine asking an AI to generate an image of the Founding Fathers of the United States. A diversity prompt might result in a more racially diverse image, misrepresenting the actual demographics of the time. This is the challenge researchers are tackling, and they've introduced a novel benchmark called DoFaiR (Demographic Factuality Representation) to evaluate the trade-off between diversity and factuality. DoFaiR uses fact-checked historical data to assess the accuracy of AI-generated images. Experiments with popular AI models like DALL-E 3 show a clear trade-off: diversity interventions increase representation but often at the expense of historical accuracy. To address this, the researchers developed a method called Fact-Augmented Intervention (FAI). FAI leverages factual knowledge from sources like Wikipedia to enhance image generation prompts, ensuring the AI is grounded in reality. The results are promising: FAI significantly improves the historical accuracy of AI-generated images while preserving diversity. This research highlights the importance of considering factuality when designing AI models, paving the way for more responsible and historically accurate text-to-image generation.