Imagine a world where chatbots could be manipulated by subtly altering their chat history. Researchers have recently explored this intriguing vulnerability in large language models (LLMs) like ChatGPT and Llama-2. These models, designed for interactive conversations, rely on chat history as context. However, they can't distinguish between real and injected history, opening the door to tampering. This research introduces a clever method to inject false information into a chatbot's memory without needing access to the model's internal workings. The trick lies in crafting special prompt templates that structure the fake history in a way the LLM interprets as genuine. To find these effective templates automatically, the researchers developed a tool called LLM-Guided Genetic Algorithm (LLMGA). This tool uses another LLM to generate and refine templates, essentially using AI to hack AI. The results are striking. By tampering with chat history, researchers could influence the chatbot's behavior, even increasing the success rate of eliciting disallowed responses up to 97% on ChatGPT. This vulnerability raises concerns about the security and trustworthiness of LLMs in real-world applications. While there are potential countermeasures, such as input and output filtering and improved safety training, the core problem lies in the LLM's inability to differentiate between user input and system context. This research highlights the need for more robust architectures that can handle different input levels separately, preventing this type of cross-contamination. As LLMs become increasingly integrated into our lives, understanding and addressing these vulnerabilities is crucial for ensuring their safe and responsible deployment.