Large language models (LLMs) have made incredible strides, but their monolithic design presents challenges for scalability, cost, and customization. Imagine trying to make a single, giant Swiss Army knife handle every possible task—it becomes unwieldy and inefficient. This research explores a more elegant solution: composing a system of expert LLMs, much like assembling a toolbox with specialized tools for specific jobs. This approach, called Composition of Experts (CoE), involves a “router” that intelligently selects the most appropriate expert LLM for a given task. This dynamic delegation allows for more efficient use of resources and potentially better performance than using a single, massive model. However, building such a system presents its own set of challenges. How do you train the router to reliably choose the right expert? How do you manage the complexities of coordinating multiple models? The researchers tackle these hurdles by proposing a two-step routing process. First, the input is classified into broad categories (like medical, legal, or coding). Then, a specialized mapping selects the best expert within that category. This modular approach offers significant advantages. Adding new capabilities, like multilingualism or specialized knowledge, becomes as simple as plugging in a new expert module. This contrasts sharply with the cumbersome process of retraining a massive monolithic model. The CoE system was implemented and tested using readily-available open-source LLMs. The results are promising, showing that a CoE can achieve comparable or even superior performance to individual expert models, especially on complex benchmarks like Arena-Hard and MT-Bench. Interestingly, the performance gains become even more pronounced when incorporating “uncertainty quantification.” This essentially means the router learns to recognize when it's unsure which expert to choose and defaults to a reliable generalist model. This research opens exciting possibilities for building more efficient, adaptable, and powerful AI systems. By moving away from the one-size-fits-all model and embracing the power of specialized expertise, we can unlock a new level of performance and flexibility in artificial intelligence. Challenges remain, particularly in ensuring the router’s accuracy and managing the computational overhead of multiple models. However, the CoE approach represents a significant step towards building more sophisticated and practical AI systems for the future.