Large language models (LLMs) like GPT-4 possess remarkable abilities to perform well on tasks they haven’t explicitly been trained on. This capability, known as out-of-distribution (OOD) generalization, is a bit of a mystery. How can these models tackle new challenges and seemingly “reason” without explicit training? New research dives into this very question, examining how LLMs achieve OOD generalization in tasks with hidden rules, similar to in-context learning scenarios. The study focuses on a specific component of the Transformer architecture called “induction heads.” Through experiments on a simplified copying task with a 2-layer Transformer, researchers made a fascinating discovery. The model exhibited a sudden shift toward OOD generalization, marked by a phenomenon called 'subspace matching' between the two layers. Essentially, the layers began to work together in a way that allowed the model to learn and apply hidden rules. Deeper investigation revealed that these two layers specialize in different aspects of the task. One processes positional information (where words appear in a sequence), while the other handles token-specific details (the meaning of individual words). Combining these abilities allows the model to learn rules and generalize to new scenarios, even when the input data looks different from what it was trained on. This compositionality, the ability to combine smaller parts into a larger whole, appears crucial for OOD generalization. The research goes beyond the simplified copying task and explores how induction heads operate in larger LLMs using more complex tasks like symbolized language reasoning (replacing words with symbols), and mathematical problem-solving using chain-of-thought prompting. Across the board, induction heads proved vital for these complex reasoning tasks. The study proposes a "common bridge representation" hypothesis, suggesting that a shared subspace in the model's inner workings acts as a central hub, connecting information from different components to facilitate reasoning and OOD generalization. This research provides a unique perspective into the fascinating inner workings of LLMs and how they solve tasks that require creative, human-like reasoning. The findings shed light on how these models perform so well on novel tasks, and they pave the way for future research into making AI models even smarter and more adaptable.