Large language models (LLMs) perform amazing feats, writing stories, translating languages, and even coding software. But beneath the surface, a fundamental question lingers: Do these digital wordsmiths truly grasp the building blocks of their creations? A new benchmark called CUTE (Character-level Understanding of Tokens Evaluation) probes the depths of LLMs' knowledge, exploring whether they understand their own tokens at the character level. The results are surprising. While LLMs excel at simple character-level tasks, such as spelling out words, like separating "hello" into "h e l l o", they stumble when faced with manipulations that humans find trivial. Take the word "international." An LLM can spell it out character by character with no problem. But ask it to insert a dash after every "n," turning it into "i-n-t-e-r-n-a-t-i-o-n-a-l," and most LLMs fail. This same pattern holds true for other tasks, like deleting, substituting, or swapping characters. The research reveals that LLMs struggle to connect their knowledge of spelling with the ability to manipulate characters in a targeted way. The research raises a crucial question: How truly deep is this understanding, and what does it imply for the future of LLMs and their ability to tackle more complex, nuanced tasks? The CUTE benchmark suggests there's still work to be done before these models can truly be said to 'understand' language in the same way humans do. This challenge also opens doors to new approaches in AI research, hinting at alternative models that might truly grasp orthography and manipulate words with the ease of a human wielding a pen.