Large language models (LLMs) are impressive, but they have a memory problem. They struggle with tasks involving lengthy texts, codebases, or datasets because their context window—the amount of information they can hold in mind at once—is limited. This bottleneck forces a trade-off between cost and performance. Processing massive amounts of data in one go is expensive, while breaking it into smaller chunks often sacrifices accuracy. Researchers have been grappling with this challenge, exploring various memory augmentation techniques, but a truly elegant and efficient solution has remained elusive. Now, a novel approach called PRISM (Processing Incrementally with Structured Memory) offers a potential breakthrough. Imagine an LLM that can “remember” and reason over vast amounts of information without breaking the bank. PRISM makes this possible by structuring the LLM’s memory using a custom schema—a blueprint that defines how key information is stored and accessed. As the LLM processes information chunk by chunk, it proposes revisions to its structured memory, gradually building a comprehensive understanding of the entire dataset. This structured approach allows PRISM to pinpoint and retain only the most relevant information, unlike previous methods that relied on verbose, unstructured summaries. This targeted memory management leads to a dramatic reduction in the amount of data the LLM needs to process, significantly cutting down on computational costs and boosting performance. Furthermore, clever caching mechanisms within PRISM allow the LLM to reuse previously computed information, leading to even greater efficiency gains. In tests, PRISM outperformed existing short-context methods on tasks like book summarization, code retrieval, and database querying, even rivaling the accuracy of long-context models while using significantly smaller chunks of data. Surprisingly, PRISM’s performance remained consistent even when the chunk size was dramatically reduced, showcasing its scalability and resilience in resource-constrained environments. One of the most exciting aspects of PRISM is its adaptability. While designing an effective memory schema requires expertise, researchers have demonstrated that LLMs themselves can be used to generate these schemas automatically. This means that PRISM can be applied to new tasks and domains with minimal human intervention. While PRISM shows immense promise, there are still challenges to overcome. Exploring the optimal design of memory schemas for different tasks and thoroughly investigating the scaling properties of PRISM will be crucial for further advancements. The ultimate goal remains to match the performance of long-context models across a wide range of tasks. PRISM takes a significant step in that direction, offering a glimpse into a future where LLMs can truly unlock the power of long-term memory.