Large language models (LLMs) have made incredible strides, yet they still struggle with nuanced language understanding, especially in less-resourced languages like Icelandic. Simply translating training data isn't enough—it misses crucial language-specific knowledge. This research explores how to enhance an LLM's performance in a specific language without sacrificing its ability to handle longer contexts, a critical factor in tasks like summarization. Researchers experimented with various "parameter-efficient fine-tuning" (PEFT) methods, essentially finding clever ways to tweak only small parts of the massive LLM, making the process faster and less resource-intensive. They discovered that strategically placing these tweaks and increasing the number of trainable parameters significantly improves language adaptation. Surprisingly, simply adding more task examples in the target language also boosts performance. The most successful approach involved a technique called LoRA, applied to specific parts of the model's architecture. This allowed the model to learn Icelandic nuances effectively without disrupting its pre-existing knowledge. However, there's a catch: some fine-tuning methods can hinder the model's ability to process longer texts, a problem researchers mitigated by focusing adjustments on the model's final layers. This research reveals crucial insights into how we can adapt LLMs for specific languages, paving the way for more efficient and effective multilingual AI.