Large language models (LLMs) are revolutionizing how we interact with technology, but their massive size presents a challenge for deployment, especially on devices with limited resources. Quantization, a technique that reduces the precision of model parameters, offers a solution by shrinking model size and speeding up inference. However, this efficiency comes at a cost: quantization inevitably leads to a drop in model quality, particularly in aggressive low-bit settings (e.g., 3-bit or 4-bit). What if we could have our cake and eat it too—maintain the efficiency of low-bit quantization while minimizing the quality loss? Researchers have explored this intriguing possibility with a novel approach called QDEC, or Quantization with Dynamic Error Compensation. The core idea is elegantly simple: store the difference between the original, high-precision weights and their quantized counterparts (the 'residuals') in the CPU’s memory. Then, during inference, dynamically fetch only the most crucial residuals to correct the most impactful quantization errors. The key innovation lies in how QDEC identifies these crucial residuals. It recognizes that not all parts of the model are equally affected by quantization. Certain channels, called salient channels, become more important due to the presence of activation outliers—unusually large activation values that amplify quantization errors. QDEC cleverly zeroes in on these salient channels at each step of the decoding process, fetching only their corresponding residuals from CPU memory. This targeted approach maximizes the quality boost while minimizing data transfer overhead, which is crucial given the relatively slower connection between CPU and GPU. This dynamic adaptation to the ever-changing landscape of activation values is what sets QDEC apart from previous methods that rely on static analysis. The results are promising. Experiments show that QDEC significantly reduces perplexity (a measure of language model quality) on standard benchmarks like WikiText, boosting the performance of 3-bit quantized models to levels comparable to their higher-precision counterparts. Furthermore, QDEC shines in challenging reasoning tasks and multi-turn conversations, demonstrating its potential to enhance real-world applications. While the improvements are less dramatic for 4-bit models (which are already quite good), QDEC still offers a noticeable edge. Perhaps most impressively, QDEC achieves these gains with minimal overhead, adding only a negligible amount to GPU memory usage and incurring a very slight slowdown in inference speed. This is achieved through careful software optimizations, including zero-copy data transfer, fast approximate Top-K selection of salient channels, and kernel fusion. QDEC offers a glimpse into the future of on-device LLM deployment, demonstrating that we can push the limits of efficiency without sacrificing performance. It paves the way for wider adoption of powerful LLMs on resource-constrained devices, bringing the transformative power of AI to a broader audience.