Large language models (LLMs) have revolutionized how we interact with and process information. But behind the scenes, optimizing these models involves intricate techniques like "whitening," a process designed to improve the quality of the data representations these models learn. However, new research suggests that whitening may not be as beneficial as previously thought, particularly for classification tasks. This research dives deep into the impact of whitening on various LLMs, revealing a surprising trend: while whitening can enhance performance in some areas, it consistently degrades the accuracy of classification tasks across different models and datasets. The study examined various models including BERT, SBERT, SimCSE, and several versions of LLaMa, finding that whitening consistently hurt classification performance, sometimes by a significant margin. The researchers observed a curious pattern: models fine-tuned for specific tasks seemed to suffer more from whitening than their general-purpose counterparts. This leads to intriguing questions about how whitening interacts with pre-trained models and those further specialized through fine-tuning. One theory is that whitening, while making features more independent, might also make it harder for the models to distinguish between different classes, crucial for accurate classification. The study’s findings provide valuable insights into optimizing LLMs. While whitening might be useful for certain tasks like semantic text similarity, it's detrimental to classification performance. This suggests that a more nuanced approach is needed, tailoring optimization methods to the specific task at hand rather than applying a one-size-fits-all solution. This work also introduces SentEval+, a new platform for evaluating LLM embeddings, allowing researchers to easily test different methods without the heavy computational demands of running full-scale LLMs. This opens doors for faster experimentation and progress in refining these powerful models.