Large language models (LLMs) are surprisingly good at math, even though they weren't explicitly taught how to do it. A new study reveals the secret behind their mathematical abilities: they use a clever technique based on Fourier features. These features allow the model to represent numbers as waves and process them layer by layer. Think of it like breaking down a sound into its individual frequencies. The research focused on how LLMs perform a simple task: addition. They found that LLMs use a two-step approach, a bit like how humans might tackle addition. First, they approximate the answer, getting close to the right magnitude (like knowing 15 plus 93 is somewhere around 100). Then, they refine it, figuring out the exact answer using modular addition, like understanding whether the answer is even or odd. What's fascinating is that different parts of the LLM work together to achieve this. The MLP layers are good at approximating the rough answer, while the attention layers do the fine-tuning with modular math. Pre-training turns out to be essential for this process. Models trained from scratch without access to a large pool of pre-existing knowledge struggle with this kind of nuanced math. This suggests that exposure to a vast amount of text during pre-training allows the model to develop these Fourier-based number representations. It's like giving an artist all the basic colors before asking them to paint – they can then mix and match to create exactly what they need. This discovery not only reveals a fundamental insight into how LLMs work but also hints at ways we might make them even better at numerical tasks. By focusing on these implicit math skills and fine-tuning how these models learn number representations, researchers may unlock further potential, allowing LLMs to solve even more complex mathematical problems.