Large language models (LLMs) like Llama-2 and GPT-4 are impressive out-of-the-box, but fine-tuning is key to tailoring them for specific tasks or datasets. However, fine-tuning all the model's parameters demands substantial computing power, making it impractical for many users. LoRA (Low-Rank Adaptation) offers a solution, enabling efficient fine-tuning with minimal performance loss. But there's a catch: recent research reveals that fine-tuning, even with benign data, can make LLMs more vulnerable to safety risks, generating harmful or inappropriate content. Enter Safe LoRA, a simple modification to the original LoRA method that enhances safety during fine-tuning. By projecting LoRA weights from specific layers onto a "safety-aligned subspace," Safe LoRA reduces safety risks without sacrificing performance. This innovative approach is data-free and training-free, requiring only the weights of the base and aligned LLMs. Experiments show Safe LoRA maintains safety levels comparable to the original aligned model when fine-tuned on malicious data. Even with mixed datasets (benign and malicious), Safe LoRA mitigates the negative impact of malicious data while preserving performance on the target task. This makes Safe LoRA a practical and effective solution for safer LLM fine-tuning, addressing a critical challenge in the rapidly evolving landscape of AI safety and accessibility.