Training massive language models like ChatGPT requires immense computing power, often involving thousands of GPUs working together. But like a team with poor communication, these GPUs spend a surprising amount of time idle, waiting for each other to share updates. This communication bottleneck significantly slows down training, hindering progress in AI. Researchers have been trying to solve this problem, but existing solutions often gobble up precious memory, making them impractical for truly large models. Now, a new technique called ACCO (Accumulate while you Communicate) is changing the game. Imagine a team learning to communicate more efficiently, overlapping conversations with their work. ACCO does just that, enabling GPUs to perform computations while simultaneously exchanging information. This clever trick hides the communication delays, making the entire training process much faster. Furthermore, unlike previous methods, ACCO manages to achieve this speed boost without hogging memory. It allows optimizer states—critical components of the training process—to be split across multiple devices. This distributed approach is like giving each team member a piece of the puzzle, allowing them to work concurrently and combine their knowledge efficiently. The secret sauce of ACCO is its two-stage mechanism. It splits the computation of updates into two parts. The first part estimates the next set of parameters, like anticipating the next move in a conversation. The second part then uses the full batch of data to calculate the actual updates, like responding effectively based on complete information. This prediction and correction process compensates for the inherent delays in parallel communication. Experiments with various LLMs, including GPT-Neo models, confirm ACCO's effectiveness, showing significant speed improvements compared to traditional methods, especially when training across multiple machines. ACCO's ability to hide communication delays and maximize hardware utilization opens exciting possibilities for training ever-larger, more powerful AI models. While it doesn’t eliminate the communication costs entirely, it significantly reduces their impact. Further research could explore moving communication processes to CPUs to free up even more GPU memory, paving the way for truly colossal models that were previously unimaginable.