Imagine teaching a child to read with a book full of typos. They might learn the misspelled words first, and then have to 'unlearn' those errors later. That's the challenge explored in new research delving into how small language models (SLMs)—compact versions of large language models (LLMs) like GPT-3—handle noise in their training data. Researchers put four SLMs, ranging from 1 to 2.7 billion parameters, to the test with five types of 'noise': flipped words, flipped characters, romanized Hindi, irrelevant information, and counterfactual statements. They wanted to see how easily these models learned the noise, how well they could unlearn it when given clean data, and if any noisy 'habits' lingered. The results showed a fascinating interplay between model size, data quality, and noise type. The smallest model, like our hypothetical child, easily picked up noisy patterns, highlighting the vulnerability of smaller AIs to bad data. Larger models showed more resilience, especially to character-level noise, suggesting that size does matter. One model, Phi2, stood out for its resistance to character-level and romanized Hindi noise. Researchers believe this robustness stems from Phi2’s unique training diet of high-quality, textbook-style data, emphasizing the importance of clean data from the start. Interestingly, all models could effectively unlearn noise when re-trained with accurate information, adapting remarkably well to their most recent 'lessons.' These findings have significant implications for AI development. They underscore the importance of data quality in training robust and reliable SLMs. Moreover, by understanding how models learn and unlearn noise, researchers can devise better strategies to immunize AIs against the inevitable imperfections in real-world data, paving the way for more resilient and trustworthy AI.