Training large language models (LLMs) to handle long contexts, like multi-turn dialogues or extensive codebases, is computationally expensive. Serving these models also presents challenges due to the massive memory requirements of key-value (KV) caches. Typically, extending context length involves a separate training stage and architectural tweaks for KV cache reduction during serving. But what if we could streamline this process? New research introduces LONGGEN, a novel approach that combines context extension with a GPU-friendly KV cache reduction architecture. This method not only reduces the training overhead but also improves long-context performance. LONGGEN's magic lies in three key insights: leveraging sparse attention patterns for efficient memory access, ensuring the model has access to all tokens through a hybrid architecture, and recognizing that lightweight training on long-context data is sufficient for significant context length extension. The results are impressive. In tests, LONGGEN achieved a 1.55x training speedup and reduced training time by 36% compared to a full-attention baseline. During inference, it slashed KV cache memory usage by a whopping 62%, resulting in significant speed improvements in both prefilling and decoding stages. What's even more remarkable is that LONGGEN outperforms baselines using traditional KV-cache reduction techniques, excelling not just in simple retrieval tasks but in complex reasoning tasks as well. This research suggests a promising path towards creating more powerful and efficient LLMs capable of handling increasingly complex, long-context tasks. As the demand for longer context processing grows, innovative approaches like LONGGEN will be essential to making LLMs more practical and accessible for a wider range of applications.