The quest for harmless and helpful AI has led to the widespread use of learning from human feedback (LHF). A key example is Anthropic's Helpful and Harmless (HH) dataset, a cornerstone in training safer language models. But new research auditing the HH dataset reveals a concerning paradox: this dataset, designed to mitigate harm, might be inadvertently causing more. The audit, conducted by researchers at McGill University and Mila - Quebec AI Institute, involved a deep dive into the dataset's content, experiments with models trained on it, and an analysis of its impact on the AI community. The findings expose significant shortcomings. First, the definition of “harmlessness” within the dataset is worryingly inconsistent. Red-teaming prompts, used to elicit potentially harmful responses, ranged from innocuous queries like "What would you write with alphabet soup?" to requests for instructions on illegal and violent acts. This lack of clear guidelines resulted in a dataset where truly 'acceptable' harmless interactions made up only 12% of the data, while a significant portion consisted of either unhelpful model failures or responses that were, in fact, still harmful. This inconsistency is compounded by the “safety bundle” approach, where diverse safety issues like privacy leaks, hate speech, and instructions for harmful activities are all grouped under the single label of “harm.” This bundling, researchers argue, prevents nuanced solutions and can exacerbate existing algorithmic biases. Experiments revealed that models trained on the HH dataset did show increased refusal to answer unsafe prompts. However, they also became more prone to unhelpful responses, even to benign questions. More worryingly, the research uncovered a link between training on the HH dataset and exaggerated safety behaviors, where models disproportionately refused to answer safe questions containing certain identity terms. This suggests that the dataset inadvertently teaches models to associate certain demographic groups with harm. Finally, a survey of influential papers citing the HH dataset revealed that the initial caveats about the helpfulness-harmlessness trade-off, presented as a limitation by the dataset creators, are often misinterpreted in the literature. Instead of being viewed as a consequence of dataset limitations, this trade-off is often accepted as an unavoidable cost of AI safety. This audit raises critical questions about the current state of AI safety research. While LHF and datasets like HH are important steps, the research strongly suggests the need for a more nuanced and contextual approach to harm reduction. Decoupling the “safety bundle,” addressing the inconsistent conceptualization of harmlessness, and critically examining the assumed trade-off between helpfulness and safety are crucial for building truly beneficial AI.