Personality plays a crucial role in how we interact with each other, and increasingly, with AI. As chatbots become more sophisticated, developers are trying to imbue them with specific personalities to make them more engaging and effective. But how do you measure a chatbot's personality? Can they even *have* one in the human sense? A new study challenges the popular practice of using standard human personality tests on chatbots, revealing some surprising inconsistencies. Researchers created 500 chatbots with distinct personality profiles based on the Big Five personality model (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) and assigned them to various tasks like job interviewing, customer service, and travel planning. They then used established personality questionnaires like the Big Five Inventory to have these chatbots essentially 'self-report' their traits. Then, they had 500 humans interact with the chatbots and rate their perceived personalities. The results were intriguing. While the chatbots' self-reported personalities showed internal consistency (meaning they answered similar questions similarly), they didn't always align with how humans perceived them, especially across different tasks. For example, a chatbot programmed to be agreeable might score high on agreeableness scales but come across as less agreeable in a job interview scenario, possibly due to the formal context. This mismatch raises serious questions about the validity of using self-report questionnaires on chatbots. Moreover, the self-reported personalities weren't very good at predicting how much humans enjoyed interacting with the chatbots. In other words, a chatbot claiming to be highly conscientious wasn't necessarily better at delivering a positive user experience. The researchers suggest that rather than relying on static questionnaires, we need to evaluate chatbot personalities dynamically, within the context of specific tasks and interactions. Just like humans express their personalities differently depending on the situation, chatbot personalities might emerge more clearly through their actions and conversational patterns than through self-assessment. This shift towards task-based evaluation could lead to the development of chatbots that are not just programmed with personalities but can genuinely express them in ways that resonate with humans, improving our overall experience with AI.