Large language models (LLMs) have become ubiquitous, powering everything from chatbots to content creation. But how do we make them even better, aligning them more closely with human preferences? A groundbreaking research paper, "Getting More Juice Out of the SFT Data," introduces a novel approach: reward learning from human demonstrations. Traditionally, LLMs are fine-tuned through supervised fine-tuning (SFT) using human-provided examples. However, this method can be limited by the quality and quantity of available data. The researchers argue that even without explicit preference data, human preferences are implicitly present in demonstration data. Their key innovation lies in leveraging inverse reinforcement learning (IRL) to extract these implicit preferences. Instead of directly learning from demonstrations, they simultaneously build a reward model and a policy model. This reward model acts as a proxy for human preferences, guiding the LLM toward generating higher-quality outputs. The paper presents two algorithms: one learns the reward model explicitly, while the other learns it implicitly. Surprisingly, the implicit approach connects to the recent self-play fine-tuning (SPIN) method, offering a new perspective on its effectiveness. The results are impressive. Experiments on various benchmarks show significant performance improvements over standard SFT. For instance, on the HuggingFace Open LLM Leaderboard, the proposed methods boost performance significantly. This research opens exciting new avenues for LLM alignment. By learning rewards directly from demonstrations, we can unlock the full potential of existing data, leading to more helpful, harmless, and aligned LLMs. While computational costs remain a challenge, the potential benefits for future AI development are substantial.