The world is a symphony of languages, but the realm of large language models (LLMs) has often been a predominantly English-speaking affair. This has created a significant gap in access to advanced AI capabilities for a vast portion of the global population. New research tackles this challenge head-on, exploring how to make preference optimization, a crucial step in LLM training, truly multilingual. Preference optimization, often involving techniques like Reinforcement Learning from Human Feedback (RLHF), helps align LLMs with human preferences, making them more helpful and effective. However, the scarcity of high-quality preference data in languages other than English has been a major roadblock. The researchers dive deep into this issue, developing innovative methods for generating and utilizing multilingual feedback data. Their experiments reveal exciting findings. Firstly, preference optimization exhibits cross-lingual transfer—training on English data improves performance in other languages, and adding more languages amplifies this effect. Secondly, adding more languages to the training data boosts overall multilingual performance, emphasizing the importance of diverse linguistic input. The study also compares different preference optimization techniques, including online and offline methods. They found that online optimization, specifically using a method called RLOO, generally outperforms offline techniques, particularly in facilitating cross-lingual transfer. The results are impressive. The team successfully preference-trained an 8-billion parameter Aya 23 model, demonstrating significant improvements over the base model and other widely-used open-source models in 23 languages. This breakthrough brings us closer to a future where AI can understand and interact with the world in its multitude of languages, unlocking AI's potential for a truly global audience. While this research marks a significant leap forward, the journey towards truly inclusive, multilingual AI is ongoing. The current model covers 23 languages, a fraction of the world's linguistic diversity. Future research will aim to expand language coverage and address challenges like dialect variations and potential cultural biases stemming from training data sources. The ongoing pursuit of multilingual preference optimization is paving the way for a more inclusive AI landscape, where language is no longer a barrier to accessing the transformative power of artificial intelligence.