Imagine training a dog meticulously on safety commands, only to find it forgets everything after learning a new trick. This 'catastrophic forgetting' is a real problem in AI, particularly with large language models (LLMs). New research reveals that when LLMs are sequentially fine-tuned, safety training is more easily forgotten than other skills. This 'biased forgetting' raises serious concerns about deploying AI safely in the real world. Researchers from Meta explored this phenomenon by training LLMs on a safety task, like identifying toxic language, followed by a capability task, such as question answering. They discovered that the safety training degraded significantly after the second task, while the capability skills remained relatively intact. This is true even with relatively benign capability training. Interestingly, reversing the order – capability training first, then safety – resulted in much less forgetting of the safety protocols. Even more concerning, this forgetting isn't uniform. Certain demographic groups mentioned in safety datasets were disproportionately affected, leading to uneven safety performance and the potential for biased outputs after capability tuning. This poses a significant challenge for building fair and equitable AI systems. This 'biased forgetting' appears linked to how the model learns. Safety tasks tend to converge to what researchers call “sharper minima” in the model’s learning landscape. These sharper minima are more susceptible to disruption during subsequent training. Conversely, capability tasks often settle into “wider minima,” making them more robust. So, what can be done? The research suggests a few promising avenues. First, re-introducing even a small amount of the original safety data during later training stages can significantly restore forgotten safety knowledge. Second, carefully choosing the order of training tasks can mitigate forgetting. Prioritizing safety training early in the process seems crucial. Finally, adjusting the learning rate – a key parameter in how AI models learn – can also influence forgetting, offering another potential control lever. This research highlights a critical challenge for developing reliable and ethical LLMs. As AI becomes more integrated into our lives, understanding and addressing this forgetting problem will be essential for ensuring safety and fairness for everyone.