Large language models (LLMs) are revolutionizing how we interact with technology, but a concerning question lingers: how secure is the data used to train these powerful AI systems? New research reveals that the risk of private information leaking from LLMs may be significantly higher than previously thought, challenging current methods of evaluating data security. Researchers from Cornell University and NVIDIA delved into the sequence-level probabilities of data leakage, uncovering surprising insights about how different factors contribute to the vulnerability of training data. They discovered that current methods, which rely on measuring the overall “extraction rate,” drastically underestimate the risk posed by randomized LLMs. This is because randomized LLMs, like those commonly used in production, can produce different outputs for the same prompt, making it harder to assess leakage using traditional methods. The study also revealed that larger models aren't always more secure. While larger models and longer input prefixes generally extract more data on average, a significant portion of individual data sequences are actually *easier* to extract from smaller models or with shorter prefixes. This challenges the assumption that simply scaling up model size automatically enhances security. Surprisingly, partial data leakage, where only fragments of information are revealed, is not as prevalent as previously thought. Even when allowing for errors in the extracted data, the researchers found that most target sequences are harder to extract partially than in their entirety. The study also confirmed that extracting later parts of a sequence is considerably easier than extracting earlier parts, highlighting the importance of considering the position of sensitive data within the training set. These findings emphasize the need for a more nuanced approach to LLM security. Instead of relying solely on average extraction rates, analyzing leakage risks at a per-sequence level provides a much more accurate picture of data vulnerability. As LLMs continue to evolve, understanding these vulnerabilities and developing robust mitigation strategies will be crucial to ensuring the responsible and ethical deployment of this transformative technology.