Large language models (LLMs) are often fine-tuned in two stages: supervised fine-tuning (SFT) for task-specific performance, followed by preference alignment (like DPO) to match human preferences. However, this sequential approach often leads to an "alignment tax," where preference alignment can degrade the performance gains achieved during SFT. A new research paper introduces PAFT (Parallel Training Paradigm for Effective LLM Fine-Tuning), a novel approach that trains SFT and preference alignment in parallel, using the same pre-trained model on their respective datasets. The models are then merged using parameter fusion. This avoids the trade-off between performance and alignment seen in sequential training. The researchers discovered that preference alignment tends to create sparse models, while SFT generates dense models. To merge these effectively, they introduce an "interference resolution" method: sparsifying the SFT model's delta parameters (the changes from the pre-trained model). This reduces redundancy and improves the merged model's performance. They tested various merging methods like TIES, Task Arithmetic and others, finding that TIES delivers best results when tested on mistral-7B. Results on benchmarks like the Hugging Face Open LLM Leaderboard and AlpacaEval demonstrate PAFT's effectiveness. A PAFT-trained 7B model achieved the top rank in its category on the Open LLM Leaderboard and topped other leaderboards overall! This parallel approach minimizes the alignment tax, allowing developers to maintain high task performance while keeping LLMs aligned with human preferences. The study also emphasizes the importance of sparse model integration and the robustness of L1-norm sparsity for various model merging techniques. Future research could explore how sparsity helps merging and tackle the challenge of updating models in production without catastrophic forgetting.