Influence functions, a tool to measure the impact of individual training data points, have been a hot topic in machine learning. But do they hold up when applied to the massive scale of Large Language Models (LLMs)? Recent research dives deep into this question, exploring whether these functions are truly effective in understanding LLM behavior. The surprising results reveal that influence functions struggle in many LLM settings. One of the primary culprits? The sheer size of LLMs makes accurately calculating influence computationally expensive, leading to approximations that can skew results. Another factor is the uncertain convergence state during fine-tuning—essentially, the model's parameters might not have settled into a stable state, making it hard to pinpoint the true influence of any single data point. Perhaps the most fundamental issue lies in the very definition of influence functions. They measure changes in model parameters, but these changes don't always correlate with changes in the LLM's actual behavior. Different parameter sets can sometimes lead to very similar outputs, making it misleading to rely solely on parameter shifts. This research highlights a critical need for new methods to understand how training data affects LLMs, opening doors for alternative approaches like representation-based methods that may offer more reliable insights into these powerful models.