Large Language Models (LLMs) have revolutionized how we interact with technology. But fine-tuning these massive models for specific tasks, especially when privacy is paramount, presents a significant hurdle. Federated Learning (FL) offers a collaborative training solution where models learn from decentralized datasets without direct data sharing. However, current methods using Low-Rank Adaptation (LoRA) in FL face limitations. They struggle to capture the full richness of diverse datasets due to constrained subspace learning. This bottleneck hinders the model's ability to effectively learn from the combined knowledge of all participants. Research reveals a more effective approach: direct weight averaging combined with a low-rank gradient optimizer like GaLore. This method allows for more efficient knowledge sharing across distributed datasets and improves performance compared to LoRA-based techniques, while using less memory for optimization during local training steps, as observed in both text-based and image-based tasks. This breakthrough opens doors for training LLMs on sensitive data like medical records or financial transactions, unlocking their full potential in a privacy-preserving manner while enhancing performance. Further research will focus on refining aggregation strategies and developing even more efficient methods for federated fine-tuning, ultimately aiming to democratize access to powerful AI while safeguarding user data.