Training massive language models (LLMs) is a computationally expensive endeavor, and figuring out the best mix of training data is even more complex. Imagine having to retrain an LLM from scratch every time you tweaked the data recipe! Researchers at Allen Institute for AI have explored a clever shortcut: what if you could train smaller models on chunks of data and then combine them to predict how a larger, fully trained model would perform? This research introduces a new method for approximating the performance of LLMs trained on different data mixes by training individual, smaller models on subsets of the data and then merging their parameters. This “mix-and-merge” strategy allows for efficient simulation of numerous data combinations without the need for full retraining each time. The researchers found that by averaging the parameters of these smaller models, they could reliably predict the performance of a single large model trained on the combined data. This approach dramatically cuts down on the computational costs, making it possible to experiment with far more data recipes and potentially discover optimal data blends that significantly improve LLM performance. The implications of this research are substantial. By drastically reducing the training costs associated with data experimentation, this method could unlock faster development cycles, enabling researchers to fine-tune the data diets of LLMs and create models that are more accurate, adaptable, and efficient. While the study primarily focused on smaller models and carefully curated datasets, the initial results are promising and pave the way for future research exploring the applicability of this approach to larger models and more diverse datasets. This could lead to a deeper understanding of the role different types of data play in shaping the behavior of LLMs. While promising, this method has some limitations. The research relies on models sharing an initial “seed” training phase. Further investigation is needed to determine the minimum shared training required for the method to be effective. Additionally, more research is needed to explore the optimal proportions of data within a mixture. Finally, further research with much larger models is needed to confirm the method's effectiveness at scale. Despite these limitations, this mix-and-merge approach provides a new tool to optimize data mixes. This work potentially changes the way we train and refine LLMs, ultimately leading to more powerful and efficient AI systems.