Large language models (LLMs) have taken the world by storm, demonstrating an impressive ability to generate human-like text, translate languages, write different kinds of creative content, and answer your questions in an informative way, even if they are open ended, challenging, or strange. One of the more intriguing mysteries of LLMs is their surprising knack for "in-context learning" (ICL). Feed an LLM a few examples of a task, and suddenly, it seems to "get it," performing the task reasonably well without any explicit training. But what's *really* going on behind the scenes? Are LLMs genuinely learning from the examples, or is something else at play? A new study suggests that memorization may be far more critical to ICL's success than we previously thought. Researchers investigated how LLMs surface memorized training data during ICL and its correlation with downstream task performance. Their findings reveal a fascinating link between memorization and performance. In many cases, even providing just a handful of demonstrations in the input prompt significantly boosts the surfacing of memorized training data. Interestingly, the most effective component in prompting the model to reveal memorized information wasn't the labels or instructions, but rather the examples themselves. This discovery raises an essential question: are LLMs really generalizing from the examples provided during ICL, or are they simply regurgitating memorized material? The study's most striking revelation lies in the strong correlation between performance and memorization. When ICL outperforms zero-shot learning (where no examples are provided), the memorization level observed in the LLM was substantially higher, often reaching 40% or more. This suggests that while LLMs may appear to be learning new tasks, their success could largely be attributed to recognizing and recalling relevant information from their massive training datasets. This new understanding of the role of memorization in ICL has significant implications for future LLM development. It points towards a critical need to explore the balance between learning from limited in-context data and leveraging the vast knowledge already stored within the model. In conclusion, while the mystique of LLM intelligence remains partially unsolved, we are gradually moving closer to a more nuanced picture of their inner workings. Further research into memorization and its connection to in-context learning is undoubtedly crucial to achieving a complete understanding of how LLMs learn.