Training large AI models like Transformers is computationally expensive, often relying on adaptive optimizers like Adam. But what if we could achieve similar or even better performance with a simpler, faster approach? New research suggests exactly that. A novel technique called SGD-SaI (Stochastic Gradient Descent with Scaling at Initialization) challenges the very need for adaptive methods. Instead of constantly adjusting learning rates throughout training like Adam, SGD-SaI analyzes the initial gradient “signal-to-noise ratio” (g-SNR) of different parameter groups within a model. This essentially assesses how noisy or sparse the gradients are for various parts of the network. Based on this initial analysis, SGD-SaI pre-conditions the learning rate for each parameter block. Then, it trains using standard SGD, without needing to continuously recalculate adjustments. This greatly simplifies the process and significantly speeds up training. Experiments show that SGD-SaI matches or exceeds the performance of AdamW in several critical areas, including image classification with Vision Transformers (ViT), language model pretraining with GPT-2, and even specialized fine-tuning tasks. Surprisingly, even with simpler models and datasets like ResNet18 on CIFAR-10, SGD-SaI demonstrates robustness and higher peak accuracy. Importantly, it achieves these results while using substantially less memory than AdamW, particularly crucial for the ever-growing size of AI models. This innovative approach could revolutionize how we train large AI models, making the process faster, more efficient, and more accessible with broader applicability. The ability to train large models with simpler methods like SGD opens doors to new research and applications, limited only by imagination. While further research with even larger models is needed, SGD-SaI promises to accelerate AI innovation by simplifying and speeding up one of its most computationally demanding aspects.