Training large language models (LLMs) like the ones powering ChatGPT is a computationally expensive affair, demanding vast resources and energy. But what if we could drastically reduce those costs without sacrificing performance? New research introduces SLTrain, a clever technique that blends "sparse" and "low-rank" approaches to make LLM pretraining significantly more efficient. Think of it like this: imagine trying to build a huge, complex Lego structure. The traditional approach (full-rank training) uses every single brick possible, leading to a massive, resource-intensive build. Low-rank training, on the other hand, tries to simplify the structure by using fewer unique brick types, kind of like using pre-fabricated sections. It's efficient, but the resulting structure might lack detail and flexibility. SLTrain takes a hybrid approach. It cleverly combines the low-rank method with a sparse strategy. This is like using a smaller selection of bricks overall (sparse), but keeping enough unique types (low-rank) to capture the essential details. The result? A near-perfect replica of the original, built with far fewer resources. SLTrain allows models to retain a high rank (meaning complexity and expressiveness) despite using fewer parameters. The secret lies in efficiently handling the sparse component by randomly fixing the "support" (the active connections within the model) and learning only the values associated with those connections. This avoids the need for complex calculations on the GPU and keeps the memory footprint low. Experiments showed impressive results: up to a 73% reduction in memory usage when pretraining the LLaMA 7B model, all without significant performance loss. When pretraining smaller models, SLTrain achieves similar performance to existing methods while lowering both parameter size and memory usage. SLTrain isn't just a theoretical breakthrough; it's highly practical. It integrates well with existing memory-saving techniques like quantization, and it's agnostic to the optimizer used during training. SLTrain could be a game-changer for training larger LLMs, opening doors for researchers and companies with limited resources to participate in the development of cutting-edge AI. Future research could focus on theoretical guarantees and exploring combinations with other structured approaches, potentially unlocking even greater efficiency gains.