Large language models (LLMs) like ChatGPT are revolutionizing how we interact with technology. But behind the scenes, efficiently serving these powerful AI models is a complex challenge. Imagine a relay race where runners seamlessly hand off the baton to maintain speed and momentum. That's the core idea behind "Baton," a new technique designed to supercharge LLM inference efficiency. Traditional LLM inference methods often suffer from idle computations. Think of it like a factory assembly line where some stations sit idle while waiting for others to catch up. Baton tackles this problem with dynamic re-batching. As one query completes its inference process, Baton immediately slots in a new query, minimizing downtime and maximizing resource utilization. The magic lies in Baton’s clever approach to vector shaping and embedding. These techniques allow it to seamlessly integrate new queries into an ongoing batch without disrupting the flow or requiring extra resources. This allows existing queries and a new query to be grouped together for efficient calculation, resulting in up to a 1.75x throughput improvement compared to previous methods. This innovative approach not only accelerates inference but also offers the potential for preemptive scheduling and flexible batch size scaling, adapting to real-time demands and preventing memory overflow. Baton is a promising step toward making LLMs faster and more accessible for everyone.