Large language models (LLMs) are impressive, but they have a big appetite for memory, especially the part called the KV cache. This cache stores past calculations to speed up processing, but it grows with longer text and bigger batches, becoming a bottleneck. Existing solutions often involve retraining models or complex token discarding strategies, which can be cumbersome and inefficient. Researchers have introduced LoRC, a new approach that uses a 'low-rank compression' technique for the KV cache, working directly with the model's weight matrices. LoRC cleverly exploits the inherent structure of these matrices, which often contain redundant information. By approximating these matrices with lower-rank versions, LoRC shrinks the memory footprint without discarding crucial data. Recognizing that LLMs have varying sensitivities to changes across their layers, LoRC implements a 'progressive compression' strategy. It analyzes the sensitivity of each layer and applies more aggressive compression to the less sensitive ones, minimizing error propagation. This means shallower layers get compressed less, preserving vital information, while deeper layers can handle more aggressive compression. Tests on popular LLaMA models across various tasks—reasoning, reading comprehension, and summarization—show LoRC significantly reduces memory needs while maintaining performance. This plug-and-play method, avoiding model retraining or complex analysis, makes deploying powerful LLMs on resource-constrained devices more feasible. This innovation paves the way for wider access to powerful AI tools, even with limited hardware resources. LoRC is a promising step towards more efficient and accessible large language models.