Large language models (LLMs) are impressive, but they sometimes "hallucinate," generating incorrect or nonsensical information. How can we tell when an LLM is making things up? Researchers have been working on ways to quantify the uncertainty of LLM outputs—essentially, to measure how confident the AI is in its own answers. A new benchmark called LM-Polygraph aims to consolidate and standardize these efforts, providing a suite of tools and evaluation techniques to compare different uncertainty quantification (UQ) methods. LM-Polygraph examines how well various UQ methods detect low-quality outputs in tasks like question answering, machine translation, text summarization, and even multilingual fact-checking. The benchmark found that simple methods, like checking the probability of the most likely answer sequence, can be surprisingly effective for short outputs. But for complex tasks, more advanced techniques based on the "diversity" of sampled answers work better. This new benchmark not only helps researchers compare UQ methods, it also helps towards creating safer and more reliable LLMs that we can trust to separate truth from fiction.