Large language models (LLMs) have revolutionized how we interact with machines, but their ability to reason and provide helpful responses can sometimes fall short. Researchers at Databricks are exploring an innovative approach to refine this process through "Critique-out-Loud (CLoud) reward models." Imagine an LLM not just answering your question, but also voicing its internal critique of its own answer before delivering it to you. That's the essence of CLoud. Traditionally, reward models used in reinforcement learning from human feedback (RLHF) act as a proxy for human preferences. They predict how much a human would like a given response, but they do this implicitly. CLoud reward models, however, add a crucial step: they first generate a natural language critique of their response, analyzing its strengths and weaknesses, before predicting a final reward score. This explicit reasoning offers several advantages. The research demonstrates that CLoud models significantly improve performance on established benchmarks like RewardBench. On both smaller (8B parameters) and larger (70B parameters) models, CLoud increased accuracy in correctly ranking preferred responses. The results also show a "Pareto improvement" in win rates, meaning the model generates higher-quality responses when using a 'best-of-N' selection strategy. For instance, out of 16 generated answers, the one chosen by CLoud was consistently better. One key element of CLoud is its 'on-policy' training. This means the model is trained on its own self-generated critiques rather than relying on external feedback. The study demonstrates the importance of this approach – models trained off-policy, using externally generated critiques, saw a significant drop in performance. The researchers further explored how to enhance the critique process by leveraging “self-consistency”– a technique where multiple critiques and reward predictions are generated before selecting the best answer. While this enhances reasoning in specific cases, it does not benefit all tasks, a finding that highlights how important understanding the specific problem context remains in LLM development. CLoud reward models could change how we train and refine LLMs by opening up more avenues for improvement by examining not only the response generated, but also the model's reasoning process itself. It’s like giving the LLM a chance to reflect, learn, and improve its own critical thinking, ultimately creating a more helpful, more nuanced, and more reliable AI experience.