Large language models (LLMs) like ChatGPT have shown impressive capabilities, but they're also vulnerable to manipulation. One particularly tricky attack is "prompt injection," where carefully crafted text can trick an LLM into ignoring its safety training and generating harmful content. Think of it like social engineering, but for AI. Researchers are exploring how to defend against these attacks, and one promising avenue is In-Context Learning (ICL). ICL allows LLMs to learn from examples provided directly in the prompt, potentially teaching them to recognize and resist malicious instructions. This paper investigates how a simple ICL technique using adversarial examples can build up LLM defenses. It turns out that by including examples where harmful requests are met with refusal, LLMs become more resistant to new injection attempts. This is like showing a child examples of bad behavior and how to say no. However, there's a catch: like an overprotective parent, this approach can lead to "over-defense," where the LLM becomes too cautious and starts refusing even harmless requests. Imagine being so afraid of scams that you refuse to answer any phone calls. The research shows promising results, with ICL significantly improving the defense against prompt injection. However, the challenge of over-defense highlights the ongoing struggle to make LLMs both safe and useful. Future work needs to refine these techniques, striking a balance between security and functionality to unlock the full potential of LLMs while mitigating the risks.