Large language models (LLMs) are getting smarter every day, but are they safe? New research explores a sneaky way to trick AIs into saying harmful things, even when they're designed not to. This trick, called a "multi-turn dialogue coreference attack," uses subtle references in conversation to bypass the AI's safety mechanisms. Imagine an AI refusing to insult someone directly. Now, imagine subtly changing the wording, referencing the person indirectly in a multi-turn conversation. Researchers found this method can trick even advanced LLMs like LLaMA2 into generating harmful responses up to 56% of the time. This research introduces "CoSafe," a new dataset of 1,400 multi-turn conversations designed to test these vulnerabilities. They tested five popular LLMs and discovered that while some AIs were more resistant than others, none were completely immune to these attacks. The study also shows that standard safety measures, like system prompts and chain-of-thought prompting, help reduce these risks but don't eliminate them completely. This raises a red flag about the safety and trustworthiness of current AI models, especially in conversational settings. While the research highlights the vulnerabilities of LLMs, it also paves the way for developing stronger safeguards. One limitation of the study is the potential for meaning to drift during these multi-turn conversations. Also, generating these complex test scenarios is currently quite expensive. The researchers hope to develop more affordable methods in the future, potentially using LLMs themselves to generate these tests.