Large language models (LLMs) have shown impressive performance on various reasoning tasks, including analogical reasoning. But are they truly grasping the underlying concepts, or are they relying on shortcuts and superficial similarities in the data they've been trained on? New research challenges the robustness of LLM analogical reasoning by testing them on modified versions of existing analogy problems. These variants maintain the core abstract relationships but alter superficial features, essentially testing whether LLMs can transfer their learned reasoning abilities to novel scenarios. The research explored three key analogy domains: letter strings, digit matrices, and stories. In the letter-string tests, LLMs struggled when the alphabet was shuffled or replaced with symbols, while human performance remained robust. Similar patterns emerged with digit matrices: LLMs faltered when the position of the missing element was changed, suggesting a reliance on positional cues rather than true pattern recognition. Story analogy tests revealed further LLM limitations, with performance impacted by answer order and paraphrasing of the stories. While LLMs initially seemed adept at identifying analogous stories, this research highlights their vulnerability to surface-level changes, unlike human reasoners who grasp the deeper causal connections. This work emphasizes the importance of evaluating LLMs beyond standard benchmarks, using variations that truly test their ability to generalize and reason abstractly. The findings suggest that while LLMs can mimic human-like reasoning in some cases, they often fall short of true analogical understanding, relying on memorized patterns and shortcuts rather than genuine cognitive flexibility.