Reinforcement Learning from Human Feedback (RLHF) is a popular technique for training large language models (LLMs) to align with human values. However, the combination of complex LLMs with the iterative nature of RLHF makes for resource-intensive training with intricate data flows. Existing systems grapple with either inflexibility in adapting to different RLHF algorithms or inefficiency in execution due to high overhead and rigid model placement. Introducing HybridFlow, a novel framework that combines the best of both worlds. By employing a hierarchical hybrid programming model, HybridFlow brings together the flexibility of single-controller management for inter-node communication and the efficiency of multi-controller execution for intra-node computation. This innovative approach streamlines the development of various RLHF algorithms, allowing researchers to switch between algorithms like PPO, ReMax, and Safe-RLHF with minimal code changes. At the heart of HybridFlow lies the 3D-HybridEngine, designed to optimize the most computationally demanding part of RLHF: actor model training and generation. This engine enables zero memory redundancy and minimizes communication overhead when switching between training and generation stages, even when using different 3D parallelism strategies. Combined with an auto-mapping algorithm to determine optimal device placement for each model, HybridFlow maximizes resource utilization and accelerates training. Experimental results showcase HybridFlow's prowess, demonstrating a significant performance boost over existing systems. On a cluster of 128 GPUs, HybridFlow achieves an impressive speedup of 1.53x to a staggering 20.57x compared to state-of-the-art baselines, making it a powerful tool for scaling LLM training and pushing the boundaries of AI.