Large language models (LLMs) possess a remarkable ability to recall facts, but how do they pinpoint the origins of this knowledge within their massive training datasets? Researchers are exploring this question through a technique called training data attribution (TDA). Imagine an LLM correctly stating that Carleton College is in the USA. TDA aims to identify the specific snippets of text from the LLM's training data that contributed to this knowledge. The challenge? These datasets can contain hundreds of billions of words, making the search for influential training examples computationally demanding. In a new study, researchers have developed a refined TDA method called TrackStar that scales to work with large LLMs and massive datasets. TrackStar uses a combination of clever techniques—including an enhanced method of filtering out noise in the model's internal representations and selectively amplifying signals relevant to a specific fact—to efficiently locate influential training examples. The researchers tested TrackStar by challenging an 8-billion-parameter LLM to answer factual questions from the T-REx dataset. They then used TrackStar to trace these answers back to their origins in the vast C4 dataset, which contains over 160 billion words. The result? TrackStar was significantly better than previous methods at identifying influential training examples, showing its promise for unlocking the secrets of LLM knowledge acquisition. Intriguingly, the research revealed a key difference between simply *finding* a fact in the training data and identifying the passages that truly *influence* the LLM's output. Traditional search methods like BM25 were better at locating passages containing the fact, but these weren’t always the snippets that shaped the LLM’s response. TrackStar, however, excelled at identifying the passages that causally impacted the model’s behavior. This suggests that LLMs may rely on more than just explicit mentions of a fact; they also leverage broader contextual information and subtle relationships within the training data. Further analysis showed that as LLMs become larger and are trained on more data, their reliance on directly entailing passages increases. This hints that larger models reason more explicitly and directly, mirroring human-like reasoning to some degree. This research opens exciting avenues for making LLMs more transparent and understandable. By pinpointing the origins of their knowledge, we can better debug their errors, refine their training data, and ultimately, build more reliable and trustworthy AI systems. The journey into the mind of an LLM is just beginning, and techniques like TrackStar are illuminating the way.