Large language models (LLMs) have revolutionized how we interact with technology. They write stories, answer questions, and even generate code. But behind their impressive capabilities lies a hidden struggle: LLMs are prone to “forgetting” previously learned information, especially during fine-tuning. This poses a significant challenge when adapting these powerful models to specific tasks, such as making them follow instructions or exhibit specific behaviors. Imagine teaching an LLM to be both helpful and safe. You might fine-tune it first on helpful responses (Supervised Fine-Tuning or SFT) and then train it to avoid harmful outputs using preference learning (like Direct Preference Optimization or DPO). However, the sequential nature of this training causes a problem. As the LLM learns to be safe through DPO, it can start to forget the helpfulness it acquired during SFT. This trade-off between different training objectives is a core issue in LLM development. New research introduces innovative training methods—ALRIGHT and MAXRIGHT—to combat this “forgetting.” Instead of training sequentially, these approaches alternate between SFT and DPO. ALRIGHT uses a probability to switch between tasks, while MAXRIGHT cleverly focuses on the objective where the LLM is performing worst. The results? These alternating methods enable LLMs to achieve a better balance, excelling at both helpfulness and safety without the usual forgetting. Experiments on models like Pythia and Llama 2 show improvements in standard benchmarks and, more importantly, in real-world tests of helpfulness and safety, all while using similar computing resources as traditional methods. This research not only offers practical solutions for LLM training but also sheds light on the complex dynamics of learning and adaptation in these powerful models. By understanding how LLMs forget, we can develop strategies to enhance their long-term learning and unlock even greater potential.