Large language models (LLMs) are increasingly being used to solve complex problems, including mathematical reasoning. A recent trend involves training LLMs to learn libraries of reusable code, effectively building their own toolboxes to tackle future challenges. But is this library learning truly happening, or are other factors at play? A new study investigates two prominent systems, LEGO-Prover and TroVE, which claim to learn libraries for mathematical reasoning. LEGO-Prover aims to construct formal proofs using a library of Isabelle lemmas, while TroVE generates Python code to solve math problems, drawing from a learned library of functions. The research reveals a surprising finding: direct code reuse in both systems is remarkably infrequent. Further investigation, through careful ablation experiments, suggests that improvements in accuracy might be attributed to self-correction and self-consistency mechanisms, rather than genuine library learning. This raises important questions about the current state of LLM library learning and emphasizes the need for better evaluation metrics beyond simple accuracy. Are LLMs truly capable of learning and reusing code like humans, or are we misinterpreting how these systems solve problems? The research calls for a more rigorous examination of LLM behavior and highlights the need for more nuanced evaluation frameworks in library learning research. Future research needs to address not only whether LLMs *can* learn reusable tools, but also *how* such tools are best learned and applied, and whether direct, verbatim reuse is the most effective strategy.