Large language models (LLMs) are all the rage, but training them requires massive computational resources. This has spurred interest in smaller-scale LLMs (SLMs), which offer a more practical solution for organizations with limited budgets. But how do you optimize the training of these smaller models for maximum efficiency? New research delves into the computational bottlenecks of training SLMs, uncovering surprising insights about hardware choices, parallelization strategies, and the importance of specialized techniques like Flash Attention. Turns out, the most expensive hardware isn't always the best. For smaller models, readily available GPUs paired with Distributed Data Parallel (DDP) can deliver exceptional performance. As model size increases, however, switching to more powerful GPUs and employing techniques like Fully Sharded Data Parallel (FSDP) becomes crucial for handling larger datasets and avoiding memory issues. One key takeaway? Flash Attention, a method designed to speed up attention mechanisms, proves significantly more impactful for SLMs than for their larger counterparts. This is because attention operations are computationally demanding, especially for models with smaller hidden dimensions. Flash Attention addresses this bottleneck, allowing for faster processing and increased efficiency. These findings provide invaluable guidance for researchers and developers looking to train SLMs effectively, paving the way for more accessible and affordable AI solutions.