Large language models (LLMs) have revolutionized various fields, but their enormous size presents challenges for fine-tuning, especially in distributed settings. The demand for more training data is ever-present, yet high-quality public datasets are dwindling. This scarcity necessitates innovative training approaches that leverage decentralized private data. Traditional federated learning, while promising, struggles with the heavy computational and communication demands of LLMs. Split learning offers a potential solution by partitioning the model and offloading the primary training burden to a powerful server. However, applying split learning to LLMs is still in its early stages. Enter SplitLoRA, a novel framework that combines the parallel training benefits of federated learning with the model-splitting efficiency of split learning. SplitLoRA employs the Low-Rank Adaptation (LoRA) technique, a parameter-efficient fine-tuning method. This allows for training on even a single consumer-grade GPU, a significant advantage over traditional full-parameter fine-tuning, which can be prohibitively resource-intensive. So how does it work? SplitLoRA partitions the LLM between client servers and a central server. Client servers perform forward propagation on their local data and send the resulting activations to the central server. The central server then continues the forward pass and performs backpropagation. The resulting gradients of the activations are sent back to the client servers for their local backpropagation. This process repeats, and periodically, client-side LoRA adapters are aggregated to synchronize the model. Extensive simulations demonstrate that SplitLoRA achieves comparable accuracy to centralized fine-tuning but at a significantly faster rate. This improvement opens doors to training large language models in resource-constrained environments, empowering edge devices and fostering collaborative AI development. While SplitLoRA shows great promise, future research directions include optimizing model splitting strategies for LLMs, adapting to heterogeneous computing resources among clients, improving efficiency for resource-limited edge devices, and enhancing privacy preservation during training. These advancements will further democratize LLM fine-tuning, enabling broader participation and innovation in AI.