Imagine a world where AI can answer any question, write any story, and translate any language in real-time. That's the promise of large language models (LLMs). But behind the scenes, there's a massive computational challenge: how do you serve these colossal AI models to millions of users simultaneously? Enter P/D-Serve, a groundbreaking system from Huawei that tackles this very problem. LLMs are complex beasts. To generate text, they perform two main phases: 'prefill' (generating the first token) and 'decoding' (generating subsequent tokens). Traditionally, these phases were handled together, creating a bottleneck. P/D-Serve cleverly disaggregates these phases, allowing them to run on separate hardware. This not only speeds up the process but also allows for greater flexibility in handling different types of requests. But disaggregation introduces its own challenges. How do you coordinate thousands of processors working on different parts of the same task? How do you ensure that the prefill and decoding phases work together seamlessly, minimizing delays? P/D-Serve solves these problems through clever engineering. It intelligently groups similar requests together, optimizing the balance between prefill and decoding resources. It also uses a novel 'on-demand forwarding' system that dynamically routes requests to the most appropriate processors, ensuring that no single processor becomes overloaded. One particularly ingenious aspect of P/D-Serve is its 'block-free' data transfer. The intermediate data generated during prefill needs to be transferred to the decoding processors. This transfer can be a major bottleneck, but P/D-Serve streamlines it by managing data in a contiguous buffer, significantly reducing overhead. The results? P/D-Serve achieves a remarkable 6.7x increase in throughput compared to traditional approaches, along with improvements in latency and overall efficiency. This means faster response times and a more seamless experience for users, paving the way for even more ambitious LLM applications. P/D-Serve isn't just a theoretical concept—it's been deployed on tens of thousands of Huawei's Ascend NPUs in a commercial setting for over eight months. It's a testament to the ongoing innovation in LLM serving and a glimpse into the future of large-scale AI.