Large language models (LLMs) like ChatGPT are trained on massive datasets, raising concerns about privacy and data security. One key question is whether these models truly “forget” the data they’ve been trained on, or if it’s possible to extract sensitive information. This is where Membership Inference Attacks (MIAs) come in. MIAs try to determine if a specific data point was part of a model's training set. Recent research has yielded inconsistent results, with some MIAs succeeding while others fail. A new study from the University of Tokyo dives deep into these inconsistencies, statistically analyzing various MIA methods across thousands of experiments. They discovered that MIA effectiveness varies significantly based on factors like model size, data domain (e.g., legal text versus code), and even the length of the text. While larger models generally performed better at resisting MIAs, the study revealed a surprising “emergent” behavior where separability between trained and untrained data suddenly increases at certain model sizes. Interestingly, the researchers found that the final layer of the LLM, often used in MIAs, might be a poor choice due to lower data separability. Also, simply checking for word differences doesn't reliably predict whether data was part of the training set. Deciding the right “threshold” to classify data as member or non-member also proved challenging. The study highlights the complex relationship between LLM size, architecture, data characteristics, and vulnerability to MIAs. It suggests that current MIAs are unreliable and that understanding the nuances of how LLMs “remember” is crucial for developing robust privacy protections. This research underscores the need for continued exploration into LLM security, particularly as these models become increasingly integrated into our digital lives.