Training large language models (LLMs) demands extensive computational resources and memory. Low-precision training offers a solution, but it often comes at the cost of accuracy and stability. Researchers have introduced a novel technique called COLLAGE, a light-weight, low-precision strategy designed to address these challenges. The core idea behind COLLAGE is to use a multi-component float representation in low-precision computations. This approach compensates for numerical errors that typically occur in low-precision training, ensuring accuracy without resorting to higher-precision clones. One of the key innovations of COLLAGE is the introduction of a new metric called "effective descent quality." This metric tracks the information lost during training due to rounding errors, providing insights into the impact of different precision strategies. Experimental results demonstrate that COLLAGE significantly improves training speed and reduces memory usage without compromising model quality. For instance, when applied to a 6.7B parameter GPT model, COLLAGE achieved a remarkable 3.7x speedup and a 23% reduction in memory usage compared to standard mixed-precision methods. COLLAGE offers a promising path towards more efficient and sustainable LLM training, paving the way for even larger and more powerful language models in the future. The ability to train with increased sequence length and micro-batch size, thanks to COLLAGE's reduced memory footprint, further enhances its practical value for researchers and developers.