Large language models (LLMs) have revolutionized how we interact with technology, demonstrating impressive abilities in tasks like translation and text generation. However, these models often excel primarily in high-resource languages like English, lagging in performance for other languages. One of the biggest hurdles in expanding LLMs to more languages is 'catastrophic forgetting.' When continually trained on new languages, LLMs tend to lose proficiency in the languages they previously mastered. It's like learning a new language and forgetting your native tongue – a frustrating problem for AI researchers. A new research paper proposes a novel architecture, MoE-CT (Mixture of Experts Continual Training), to tackle this challenge. The MoE-CT approach works by separating the base model's learning from the new language acquisition process. The parameters of the original LLM are frozen, preserving its hard-won knowledge. A separate Mixture of Experts (MoE) module is then added. This MoE module is trained on the new language data, allowing the LLM to acquire multilingual capabilities without overwriting its existing knowledge. This method represents a departure from conventional continual training (CT) approaches. Traditional CT methods often involve retraining the entire model, increasing the risk of catastrophic forgetting. The MoE-CT strategy reduces the need for massive amounts of original language data during continual training. In experiments using the Qwen LLM as a base, MoE-CT demonstrably improved performance in multiple languages. Crucially, it achieved this without significant performance loss in the original languages. This advance has exciting implications for making LLMs more inclusive and globally accessible. Imagine an LLM that speaks hundreds of languages, opening up communication and information access to a far broader audience. While this research represents significant progress, further work is needed. The researchers plan to explore the efficacy of MoE-CT across other open-source LLMs. This will give a clearer understanding of whether the approach generalizes to different models. Furthermore, exploring how to best train the MoE module with larger datasets will be essential for maximizing multilingual performance. The MoE-CT architecture offers a path towards building LLMs that can learn and adapt to new languages continuously, without losing their existing expertise—a crucial step toward truly universal language technology.