Large language models (LLMs) have shown promise in complex tasks like math problem-solving and code generation. However, they often struggle with intricate reasoning. A common technique to boost their skills is preference optimization, where the model learns to prefer better solutions over worse ones. This typically relies on human-labeled data, which can be expensive and time-consuming to obtain. What if LLMs could learn to reason more effectively without constant human intervention? New research explores using "pseudo feedback" – automatically generated feedback that mimics human preferences – to optimize LLMs for reasoning tasks. Researchers explored two main types of pseudo feedback. The first leverages "frontier LLMs" – highly capable models like GPT-4 – to generate solutions and test cases. By checking if a model's solution passes the tests created by a stronger model, we can create a preference signal. The second approach uses a clever trick called "self-consistency." The idea is to generate multiple solutions from the model itself and treat the most frequent outcome as the correct answer. This is particularly helpful when access to a frontier LLM is limited. Experiments on math and coding tasks show promising results. For math, the technique improved the performance of a 7-billion parameter Mathstral model from 58.3% to 68.6% accuracy on the MATH dataset, outperforming much larger models like a 72-billion parameter NuminaMath model and even exceeding the performance of GPT-4-Turbo in some cases. In coding, similar gains were observed on benchmarks like APPs and LiveCodeBench. Interestingly, these two types of pseudo feedback work well together. Training first with frontier LLM feedback and then refining with self-consistency leads to even better performance. This research highlights an efficient way to enhance LLM reasoning abilities without the bottleneck of human-labeled data. By letting LLMs learn from themselves and from stronger models, we can unlock their potential for more complex reasoning in diverse fields. However, the research also points out that the quality of pseudo feedback becomes crucial. As models improve, the easy problems get solved, and the remaining challenging ones may lack accurate feedback, leading to a performance plateau. One solution is continually adding new, unseen problems to keep challenging the model. Further research will likely explore more sophisticated ways to generate and utilize pseudo feedback, along with combining it with other techniques like scaling inference to further push the boundaries of LLM reasoning capabilities.