Large language models (LLMs) are impressive, but they sometimes generate incorrect or “hallucinated” information. This poses a serious problem for applications requiring factual accuracy. Imagine an AI assistant confidently giving you wrong medical advice or a chatbot making false claims about historical events. That's why researchers are working hard on fact-checking mechanisms for AI-generated text. A new, lightweight approach called Provenance offers a promising solution. Unlike resource-intensive methods that rely on other LLMs for verification, Provenance uses smaller, open-source natural language inference (NLI) models. This makes it faster, cheaper, and more accessible. Here's how it works: When an LLM generates text based on some source material, Provenance first identifies which parts of the source are most relevant to the user's query. It then constructs a 'claim' from the query and the LLM’s generated answer. Finally, it uses an NLI model to check how well each relevant piece of source material supports the claim, producing an overall factuality score. This score helps determine if the LLM's output aligns with the provided context. The system has been tested on a range of datasets, including TRUE, MSMarco, TruthfulQA, HotpotQA, HaluEval, and HaluBench, covering diverse question-answering scenarios. Results show that Provenance effectively identifies hallucinations, achieving high accuracy across various tasks. While this research focuses on short answers and text-based contexts, it represents a significant step toward more reliable and trustworthy LLM applications. Future research could expand Provenance to handle longer responses, different data formats (like conversations or tables), and more complex reasoning tasks. As LLMs become increasingly integrated into our lives, solutions like Provenance will be crucial for ensuring they provide accurate and dependable information.