Published
Jun 22, 2024
Updated
Oct 11, 2024

Can AI Learn When to Say “I Don’t Know”? Multilingual Edition

Teaching LLMs to Abstain across Languages via Multilingual Feedback
By
Shangbin Feng|Weijia Shi|Yike Wang|Wenxuan Ding|Orevaoghene Ahia|Shuyue Stella Li|Vidhisha Balachandran|Sunayana Sitaram|Yulia Tsvetkov

Summary

Large language models (LLMs) are impressive, but they sometimes confidently give wrong answers, a problem called "hallucination." One solution is to teach them when to abstain, or say "I don't know." However, current techniques mostly work for English. New research tackles how to teach LLMs to abstain in multiple languages. The challenge is that these models' knowledge and reasoning abilities are often weaker in languages other than English. Researchers found that simply translating English abstention methods doesn't work well – there's a significant performance drop in other languages, particularly under-resourced ones. Their solution? Multilingual feedback. The LLM generates possible answers, then provides feedback on its own answers *in multiple related languages*. This cross-linguistic reflection helps identify gaps and inconsistencies across different perspectives. For example, if an LLM is asked a question in Bengali, it might also generate feedback in Hindi, Marathi, and Nepali. This helps catch errors an English-only or even just Bengali-only approach might miss. It turns out related languages often reveal subtle cultural and contextual differences that highlight weaknesses in the model's understanding. The results? This method significantly outperforms existing techniques, especially for lower-resourced languages. It not only improves accuracy but also leads to more equitable performance across languages. Interestingly, the researchers found that cultural factors strongly influence how LLMs abstain. Questions on Western-centric topics like U.S. history were more likely to cause inconsistencies across languages, highlighting the need for more culturally diverse training data. This research shows promising progress in making LLMs more reliable and trustworthy across languages, opening doors to broader global access to accurate information. Future research could explore how to optimize language selection for feedback and tailor strategies for specific languages and cultural contexts.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does the multilingual feedback mechanism work in teaching LLMs to abstain from answering?
The multilingual feedback mechanism is a two-step process where the LLM first generates potential answers, then evaluates these answers across multiple related languages. For example, when faced with a question in Bengali, the system would: 1) Generate initial answers in Bengali, 2) Produce feedback in related languages like Hindi, Marathi, and Nepali, 3) Cross-reference these perspectives to identify inconsistencies or gaps in understanding. This process helps catch errors by leveraging cultural and linguistic similarities between related languages. In practice, this could be applied to a medical chatbot that needs to provide accurate health information across South Asian languages while knowing when to defer to human expertise.
What are the main benefits of AI systems that can admit uncertainty?
AI systems that can admit uncertainty offer several key advantages. First, they provide more reliable and trustworthy interactions by avoiding false or misleading information. Second, they enhance safety in critical applications like healthcare or financial advice by knowing when to defer to human experts. Third, they help build user trust by being transparent about their limitations. For example, in customer service, an AI that admits when it doesn't know something and transfers to a human agent is more helpful than one that gives incorrect information. This capability is particularly valuable in professional settings where accuracy is crucial.
How can multilingual AI improve global communication?
Multilingual AI can significantly enhance global communication by breaking down language barriers while maintaining accuracy across different cultures. It helps businesses expand internationally by providing reliable translations and cultural context, enables more inclusive education by making resources available in multiple languages, and facilitates cross-cultural collaboration in global teams. For example, a company could use multilingual AI to provide customer support across different regions while ensuring consistent quality and cultural appropriateness. This technology is particularly valuable for international organizations, educational institutions, and global commerce platforms.

PromptLayer Features

  1. Testing & Evaluation
  2. The paper's multilingual feedback approach requires systematic evaluation across language pairs, which aligns with PromptLayer's testing capabilities
Implementation Details
1. Create test sets for different language pairs 2. Set up batch testing workflows 3. Configure evaluation metrics for abstention accuracy 4. Implement cross-language validation checks
Key Benefits
• Automated cross-lingual testing at scale • Consistent evaluation across language pairs • Standardized abstention accuracy metrics
Potential Improvements
• Add language-specific scoring algorithms • Implement cultural context validation • Develop automated regression testing for abstention behavior
Business Value
Efficiency Gains
Reduces manual testing effort by 70% through automated multilingual evaluation
Cost Savings
Cuts evaluation costs by identifying issues earlier in development
Quality Improvement
Ensures consistent abstention behavior across all supported languages
  1. Workflow Management
  2. The multi-step process of generating answers and cross-linguistic feedback requires sophisticated workflow orchestration
Implementation Details
1. Create templates for answer generation 2. Design feedback collection workflows 3. Set up language-specific processing steps 4. Implement result aggregation
Key Benefits
• Streamlined multilingual processing • Reproducible feedback collection • Versioned workflow templates
Potential Improvements
• Add dynamic language selection • Implement parallel processing • Create adaptive feedback loops
Business Value
Efficiency Gains
Reduces workflow setup time by 50% through reusable templates
Cost Savings
Minimizes resource usage through optimized processing chains
Quality Improvement
Ensures consistent application of abstention methods across languages

The first platform built for prompt engineering