Imagine a world where AI models could collaborate, seamlessly merging their strengths and compensating for each other's weaknesses. This isn't science fiction, it's the promise of ensemble learning, a technique gaining traction in the world of large language models (LLMs). Traditionally, ensembling has happened at the sample level (comparing entire responses) or the token level (comparing individual words or sub-words). But these methods have limitations. Sample-level approaches can miss opportunities to refine responses mid-generation, while token-level methods often get bogged down in the granular details, losing sight of the bigger picture. Researchers have introduced a new approach called SWEETSPAN, which ensembles at the *span* level, analyzing chunks of words. This middle ground aims to provide enough context for informed decisions without sacrificing the flexibility of real-time adjustments. SWEETSPAN works in two key steps. First, each candidate model independently generates spans of text based on a shared prompt prefix. Then, a clever perplexity-based system evaluates these spans, filtering out unreliable scores to ensure robust selection. The results are impressive. Across various tasks like commonsense reasoning, math problems, code generation, and machine translation, SWEETSPAN outperforms individual models and existing ensemble methods. Significantly, SWEETSPAN shines even when combining high-performing models with weaker ones, a scenario common in real-world applications. This robustness stems from its ability to filter out unhelpful evaluations, preventing underperforming models from dragging down the ensemble. While the efficiency overhead of ensemble methods remains a challenge, SWEETSPAN’s parallel processing capability and flexible span lengths offer promising avenues for optimization. Span-level ensembling represents a significant step toward unlocking the full potential of LLMs. By encouraging collaborative generation, we can build more robust, versatile, and ultimately smarter AI systems.