Large language models (LLMs) are transforming how we interact with technology, powering everything from chatbots to AI-powered search. But as these models grow more complex and context windows expand to accommodate more sophisticated tasks, serving them efficiently becomes a critical challenge. Hybrid LLMs, which combine the strengths of attention mechanisms with the efficiency of recurrent layers like State Space Models (SSMs), offer a promising path to handling long contexts. However, they also introduce unique caching challenges. Traditional caching methods, designed for standard attention-based models, struggle with the in-place state updates of recurrent layers. These updates prevent efficient rollback for partial sequence overlaps, leading to an explosion of large cache entries with minimal reuse. Enter Marconi, a groundbreaking new caching system designed specifically for the era of hybrid LLMs. Marconi tackles the caching problem head-on with innovative admission and eviction policies. Instead of relying solely on recency, Marconi’s admission policy intelligently predicts the reuse likelihood of potential cache entries based on a clever categorization of prefix reuse scenarios. By analyzing whether shared prefixes arise from purely input elements like system prompts or a combination of input and output tokens like conversation history, Marconi selectively caches only the most promising SSM states. This judicious approach dramatically reduces the number of low-utility cache entries. On the eviction side, Marconi introduces a FLOP-aware policy that considers not just recency, but also the potential compute savings offered by each cache entry relative to its memory footprint. This allows Marconi to prioritize longer, more computationally intensive sequences, maximizing the efficiency gains offered by hybrid models. The results are impressive. Across diverse workloads and hybrid model architectures, Marconi achieves up to a 34.4x increase in token hit rates, resulting in latency reductions of up to 71.1%. This translates to significantly faster response times, especially for longer contexts, higher ratios of SSM layers, and larger SSM state dimensions - trends that align with the direction of current LLM development. Marconi’s innovative approach paves the way for more efficient and responsive hybrid LLM serving, unlocking the potential of these powerful models for a wider range of applications. As LLMs continue to evolve, efficient caching systems like Marconi will be essential to making these models practical for real-world deployment.