Imagine an AI that never stops learning, constantly adapting to new information without forgetting what it already knows. That’s the dream of continual learning, and researchers are working hard to bring it to reality. Large Language Models (LLMs) have shown remarkable progress, but they encounter challenges with this type of continuous adaptation due to ‘catastrophic forgetting,’ where new knowledge overwrites previous learning. This limitation incurs considerable environmental and economic costs since retraining LLMs with new information is resource-intensive. This paper introduces Progressive Mixture of Experts with Asymmetric Transformer (PMoE), a novel approach to continual learning aiming to minimize forgetting and improve efficiency. PMoE’s key innovation lies in its asymmetric design. The model has shallow layers dedicated to preserving general knowledge and deeper layers specializing in new information. These deep layers incorporate progressively added ‘experts,’ and a ‘router’ that efficiently directs new knowledge to the appropriate expert. The router uses deep features which contain consolidated information, which improves the model’s performance and reduces the forgetting problem. Experiments conducted using TRACE datasets and general language understanding benchmarks reveal PMoE’s superior performance compared to state-of-the-art methods. It outperforms LoRA and full fine-tuning, needing fewer trainable parameters while preserving general and tuned abilities. The asymmetric design effectively preserves prior knowledge while integrating new knowledge, and an analysis of the routing mechanism shows how PMoE allocates information efficiently across its experts based on input patterns. This advance in continual learning is a step towards LLMs that retain their existing knowledge while continuously acquiring new information. This parameter-efficient and robust model opens exciting avenues for future AI applications by enabling continuous adaptation without the cost of forgetting.