Imagine an AI that not only generates text but also critiques its own work, constantly striving for improvement. This self-reflection isn't science fiction; it's the core of a new technique called Critic-RM, designed to make large language models (LLMs) more aligned with human preferences. Critic-RM works by training LLMs to generate multiple critiques for their own responses, then filtering and refining these critiques to ensure they're consistent with human judgments. This self-criticism then becomes part of the reward signal, nudging the LLM towards generating more helpful, harmless, and accurate outputs. Experiments on various benchmarks, including RewardBench and CrossEval, show Critic-RM outperforms traditional reward models, particularly in complex reasoning and safety tasks. It's even more data-efficient, achieving impressive results with limited training data. However, generating critiques adds computational overhead, posing a challenge for real-time applications. Future research could explore iterative training to further boost performance and make self-critiquing AI an even more powerful tool for improving language models.