Large language models (LLMs) are memory hogs. Their key-value (KV) cache, essential for storing past calculations and speeding up text generation, grows rapidly with the input length. This poses a significant challenge, especially for long-context applications. Quantization, a technique for reducing the precision of numbers stored in memory, offers a promising solution. But typical methods have a hidden cost: storing metadata for each quantized block creates significant overhead. This paper introduces QJL, a clever new quantization technique with *zero* such overhead. The secret sauce? QJL pairs two ideas: first, a well-known technique called Johnson-Lindenstrauss (JL) transforms projects the data onto a smaller, random dimension, like a shadow preserving essential information. Second, QJL takes the JL result and simply stores its sign (+ or -). The result is an extremely compact representation that doesn’t need to store extra metadata, unlike other quantization methods. To retrieve information, QJL relies on an asymmetrical trick: it applies JL to the incoming query, but without quantizing the result. Surprisingly, by keeping one side in full precision, they derive an accurate estimate of the original values. Even with the aggressive simplification from JL and sign-bit quantization, this approach has minimal distortion in the final attention scores, the critical values for text generation. Testing QJL across several LLMs and tasks shows up to 5x reduction in KV cache memory use *without* harming accuracy. In some long-context question answering tasks, QJL even *improves* the F1 score compared to other quantization methods. This suggests that QJL might bring some regularization benefits alongside its memory savings. What makes this particularly attractive for GPU-heavy applications is its speed and efficiency. Initial experiments with a specialized CUDA kernel highlight faster generation times, and the team plans further optimization within CUDA. While QJL effectively tackles common cases, there are some outliers. These outlier values in deeper layers are simply handled by applying QJL multiple times with different compression rates. The QJL breakthrough demonstrates an elegant approach to shrinking LLM memory with minimal impact on performance. By combining clever mathematical techniques with efficient implementation, QJL pushes the boundaries of what's possible in long-context LLM applications.