Published
Jun 25, 2024
Updated
Oct 17, 2024

Can AI Learn to Ask Better Questions? Enhancing LLMs with Information Gain

Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain
By
Davide Mazzaccara|Alberto Testoni|Raffaella Bernardi

Summary

Large language models (LLMs) are impressive, but they sometimes struggle with a fundamental aspect of communication: asking good questions. Think of playing 20 Questions—while an LLM might eventually guess the answer, its questions are often inefficient, taking far longer than a human would. This research explores a novel way to make LLMs better question-askers by focusing on what’s called 'Expected Information Gain,' or EIG. Essentially, EIG measures how much a question helps narrow down the possibilities. This work uses a clever three-step method: First, it has the LLM itself generate multiple questions. Then, it uses EIG to identify the most informative questions, effectively having the LLM grade its own work! Finally, it uses these self-graded questions to fine-tune the model using a technique called Direct Preference Optimization (DPO), where it rewards better questions. The results are intriguing. The trained model consistently asks significantly better questions, leading to quicker solutions in the 20 Questions game. More surprisingly, this improvement also works in topics the model wasn’t specifically trained on, suggesting a more fundamental improvement in its ability to formulate questions. This research opens up interesting possibilities. Imagine an AI assistant that not only understands your requests but also asks clarifying questions to pinpoint your exact needs. Or think of customer service chatbots that efficiently gather information to resolve issues faster. The challenge now is scaling this approach beyond the relatively simple world of 20 Questions. While calculating EIG becomes more complex with open-ended scenarios, this initial step shows a lot of promise for training AIs to ask effective questions and efficiently gather information in real-world situations.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does the three-step EIG-based method work to improve LLMs' question-asking abilities?
The method combines question generation, self-evaluation, and model fine-tuning to enhance question quality. First, the LLM generates multiple potential questions. Then, it uses Expected Information Gain (EIG) metrics to evaluate these questions' effectiveness at narrowing down possibilities. Finally, Direct Preference Optimization (DPO) fine-tunes the model by rewarding questions that score higher on EIG measurements. For example, in a customer service scenario, this process would help the AI learn to ask targeted questions like 'Is this a billing or technical issue?' rather than vague queries that don't effectively narrow down the problem scope.
What are the practical benefits of AI systems that can ask better questions?
AI systems with improved question-asking abilities can dramatically enhance user interactions and problem-solving efficiency. These systems can better understand user needs, reduce confusion, and reach solutions faster. Key benefits include more efficient customer service interactions, better virtual assistants, and improved learning tools. For instance, a smart home assistant could ask precise clarifying questions about your preferences when setting up routines, or a healthcare chatbot could more accurately assess symptoms before directing patients to appropriate care options.
How might better AI questioning capabilities transform customer service?
Enhanced AI questioning capabilities could revolutionize customer service by making interactions more efficient and effective. Instead of following rigid scripts, AI systems could ask intelligent, context-aware questions to quickly identify and resolve issues. This leads to faster problem resolution, higher customer satisfaction, and reduced support costs. For example, when a customer reports a problem, the AI could ask targeted questions based on previous similar cases, leading to faster diagnosis and resolution of common issues while reducing the need for human intervention in routine matters.

PromptLayer Features

  1. Testing & Evaluation
  2. EIG-based question evaluation aligns with PromptLayer's testing capabilities for measuring prompt effectiveness
Implementation Details
1. Create test sets with known optimal questions 2. Set up EIG scoring metrics 3. Configure automated testing pipelines 4. Compare different prompt versions
Key Benefits
• Systematic evaluation of question quality • Automated comparison of prompt versions • Quantifiable improvement tracking
Potential Improvements
• Add EIG-specific scoring metrics • Integrate with DPO fine-tuning workflows • Expand test case coverage beyond 20 Questions
Business Value
Efficiency Gains
30-50% faster question optimization cycles
Cost Savings
Reduced fine-tuning costs through targeted improvements
Quality Improvement
More efficient information gathering in applications
  1. Workflow Management
  2. Multi-step question generation and evaluation process maps to workflow orchestration needs
Implementation Details
1. Define question generation templates 2. Create EIG evaluation steps 3. Set up DPO fine-tuning pipeline 4. Configure version tracking
Key Benefits
• Reproducible question optimization process • Versioned prompt improvements • Automated workflow execution
Potential Improvements
• Add specialized EIG calculation nodes • Integrate fine-tuning feedback loops • Support dynamic template adjustment
Business Value
Efficiency Gains
40% reduction in workflow setup time
Cost Savings
Minimized manual intervention costs
Quality Improvement
Consistent question quality across applications

The first platform built for prompt engineering