Training large language models (LLMs) demands substantial memory, especially for optimizer states. Low-rank training methods offer memory savings but often compromise performance. A new method called Fira aims to break this trade-off by achieving full-rank training under low-rank constraints. The challenge lies in preserving the full optimizer states needed for correcting raw gradients in full-rank training when working with limited memory. Fira introduces a norm-based scaling method that leverages the scaling effects of adaptive optimizers. It has been observed that the scaling factor of an optimizer remains similar from low-rank to full-rank training. This lets Fira utilize the low-rank optimizer's scaling impact as a substitute for the full-rank optimizer. This method maintains the low-rank constraint in the optimizer while correcting gradients in full-rank training. Another obstacle is the occurrence of sudden spikes in gradients, leading to loss spikes. Fira tackles this by using a norm-growth limiter, which smooths gradient increases by regulating the growth of gradient norms. Experiments on LLaMA models demonstrate that Fira not only reduces memory usage but also achieves performance comparable to or exceeding both low-rank and full-rank training baselines. In pre-training a 1B parameter LLaMA model, Fira reduces optimizer state memory usage by 61.1% while improving performance. It also excels at lower ranks, maintaining high performance even with severe rank constraints. Fira's success in both pre-training and fine-tuning tasks suggests a potential shift in LLM training. By combining memory efficiency with high performance, Fira opens the door to more powerful, more accessible LLM training.