Can AI truly understand our feelings? That's the complex question researchers are tackling as they develop empathetic conversational systems (ECS) – chatbots designed to respond with more than just programmed politeness. A new research paper proposes a multidimensional evaluation framework for measuring empathy in these systems, moving beyond traditional metrics like helpfulness and relevance. Current methods often fall short in capturing the nuances of empathy. Simply comparing a chatbot's response to a "gold standard" human response doesn't necessarily tell us how empathetic the chatbot actually is. This new framework offers a more in-depth analysis by evaluating empathy across three dimensions: emotional reaction (affective empathy), interpretation (cognitive empathy), and exploration. Researchers tested this framework using state-of-the-art ECS models and large language models (LLMs) like GPT3.5 and Vicuna. Interestingly, LLMs often outperformed specialized ECS models in exhibiting empathetic behaviors like concern and consolation. One particularly promising area is the development of an "empathy lexicon" – a curated list of words and phrases that signal empathetic responses. This lexicon-based approach could provide a more objective measure of empathy, but it also faces challenges, as chatbots might learn to manipulate the lexicon without genuine understanding. This research signals an important shift in evaluating AI. It's not enough for chatbots to sound human; they need to demonstrate a capacity for genuine emotional connection. While much work remains in developing truly empathetic AI, this framework lays the foundation for building and assessing chatbots that go beyond programmed responses and truly connect with human emotions.