Generating text with AI is like a high-wire act—impressive, but prone to stumbles. Large Language Models (LLMs) can write stories, answer questions, and even translate languages, but how can we tell when they’re confident in their answers? Simply calculating the probability of a generated sequence often falls short, as it mixes up how grammatically correct something sounds with whether or not it's actually *true*. New research introduces “Contextualized Sequence Likelihood” (CSL), a clever method to fine-tune that probability. Imagine an LLM answering the question, “Who walked on the moon in 1969?” A response like, “Neil Armstrong took his famous first steps on the lunar surface on July 20, 1969, alongside Buzz Aldrin,” is correct but includes extra details. CSL uses the LLM’s own *attention mechanism*—essentially what the model focuses on—to weigh the important words more heavily. So, “Neil Armstrong” and “1969” get more weight than “Buzz Aldrin” or “July 20th” in determining the model's confidence. This makes CSL a much better judge of answer quality than traditional methods. Researchers tested CSL on question-answering datasets and various LLMs. Across the board, CSL was significantly better at distinguishing correct from incorrect answers. Even without prompting, CSL could often pinpoint the critical parts of a response, hinting at a deeper understanding within these models. This breakthrough has exciting potential for making LLMs more trustworthy. Imagine AI systems that know when to ask for help, admit uncertainty, or double-check their work. CSL moves us closer to this goal, making AI not just impressive, but also reliable. Further research will likely focus on making the “attention weights” more understandable to us humans and adapting CSL to different tasks beyond question-answering.