The world of Large Language Models (LLMs) is constantly evolving, with closed-source models like GPT-4 often setting the benchmark. However, open-source alternatives are catching up fast. Researchers at NVIDIA have introduced ChatQA 2, a new LLM built on Llama 3.0, boasting a massive 128K context window. This new model aims to rival proprietary LLMs in two key areas: understanding incredibly long contexts and performing retrieval-augmented generation (RAG). The ability to handle long contexts is crucial for processing large amounts of information, while RAG allows LLMs to access and use external knowledge, boosting their capabilities. The team extended Llama 3’s context window through a continued pre-training process using a mix of existing and upsampled long sequences of data. Furthermore, a three-stage instruction-tuning method refined the model's instruction-following abilities, RAG performance, and long-context understanding. The results are impressive. ChatQA 2 outperforms several state-of-the-art models, including GPT-4-Turbo, on tasks requiring ultra-long context understanding (over 100K tokens). It also excels in RAG benchmarks using a shorter 4K context window. Interestingly, the research team discovered that even powerful long-context LLMs benefit from RAG when retrieving more chunks of relevant information. With enough relevant context, RAG consistently beats direct long-context methods, even in models as powerful as ChatQA 2. This work highlights the exciting progress of open-source LLMs. By providing both long context and robust RAG abilities, ChatQA 2 offers flexibility for diverse applications, paving the way for more powerful and accessible AI solutions. The open-sourcing of the model, data, and evaluation methods further empowers the community to build upon these advancements.