Can AI Fill the Gaps in Our Knowledge?
Enriching Ontologies with Disjointness Axioms using Large Language Models
By
Elias Crum|Antonio De Santis|Manon Ovide|Jiaxin Pan|Alessia Pisu|Nicolas Lazzari|Sebastian Rudolph

https://arxiv.org/abs/2410.03235v2
Summary
Knowledge graphs, the interconnected webs of facts that power many AI systems, rely heavily on ontologies—structured vocabularies that define relationships between concepts. These ontologies, however, are often incomplete, lacking crucial information about how concepts *don't* relate. Specifically, they often miss "disjointness axioms," which specify that certain categories are mutually exclusive (like "mammal" and "fish"). Researchers are exploring whether Large Language Models (LLMs) can help automate the process of filling these gaps by identifying disjoint classes within existing ontologies. The core idea is to leverage the implicit knowledge contained within LLMs, gained from their vast training data, to deduce which concepts should be mutually exclusive. The challenge lies not only in getting LLMs to perform this classification accurately but also in ensuring that the new disjointness information doesn't introduce logical contradictions into the ontology. The researchers tested different prompting strategies on several publicly available LLMs, including Mistral, Gemma, Llama, and Qwen, using the DBpedia ontology as a testbed. They found that, surprisingly, providing minimal task description yielded better results than few-shot examples. Additionally, framing the task as a negative question (e.g., "Can a mammal be a fish?") proved more effective. While the results are promising, there's still room for improvement. Some LLMs were too "conservative," failing to identify all truly disjoint classes, while others were too "aggressive," incorrectly labeling some classes as disjoint. Future research will focus on refining prompting techniques, incorporating more contextual information, and exploring more advanced methods like Retrieval Augmented Generation to improve the accuracy and reliability of LLM-driven ontology enrichment. The ultimate goal is to harness the power of LLMs to automatically enhance our knowledge graphs, making AI systems more robust, consistent, and capable of more sophisticated reasoning.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team.
Get started for free.Question & Answers
What prompting strategies were most effective for LLMs in identifying disjoint classes within ontologies?
The research revealed two key effective prompting strategies: minimal task description and negative question framing. Rather than using few-shot examples, providing simple, direct task instructions yielded better results. The process works by formulating queries as negative questions (e.g., 'Can a mammal be a fish?') rather than positive statements. For example, in a medical ontology, the system might ask 'Can a viral infection be a bacterial infection?' to establish disjointness between disease categories. This approach helps LLMs leverage their implicit knowledge more effectively while reducing confusion from complex prompting patterns.
How can AI knowledge graphs improve everyday decision-making?
AI knowledge graphs help make everyday decisions by organizing information in an interconnected, easily accessible way. Think of them as digital maps of facts and relationships that AI can quickly navigate. For example, when planning a trip, a knowledge graph could connect information about weather patterns, local events, transportation options, and accommodation availability to help make better travel decisions. They're particularly valuable in areas like healthcare (connecting symptoms to potential diagnoses), education (linking related concepts for better learning), and shopping (connecting products with reviews, prices, and alternatives).
What are the main benefits of using AI to enhance knowledge organization?
AI-enhanced knowledge organization offers several key advantages: improved accuracy in categorizing information, faster data processing, and the ability to discover hidden relationships between concepts. It helps businesses and organizations make sense of large amounts of data by automatically identifying connections and patterns that humans might miss. For instance, in customer service, AI can organize customer feedback into meaningful categories and identify trends, while in research, it can help scientists discover unexpected connections between different studies or findings. This leads to better decision-making and more efficient information management.
.png)
PromptLayer Features
- Testing & Evaluation
- The paper's comparison of different prompting strategies across multiple LLMs aligns with PromptLayer's batch testing capabilities
Implementation Details
1. Create test sets of known disjoint classes, 2. Set up automated batch tests across different prompting strategies, 3. Compare results across multiple LLMs using standardized metrics
Key Benefits
• Systematic comparison of prompt effectiveness
• Reproducible evaluation across multiple LLMs
• Automated performance tracking over time
Potential Improvements
• Integration with ontology validation tools
• Custom scoring metrics for disjointness accuracy
• Automated regression testing for ontology consistency
Business Value
.svg)
Efficiency Gains
Reduces manual testing time by 70% through automated batch evaluation
.svg)
Cost Savings
Optimizes LLM usage by identifying most effective prompting strategies
.svg)
Quality Improvement
Ensures consistent and reliable ontology enrichment results
- Analytics
- Prompt Management
- The research's finding about minimal prompts vs. few-shot examples highlights the importance of systematic prompt versioning and optimization
Implementation Details
1. Create template library for different question formats, 2. Version control prompt variations, 3. Track performance metrics for each prompt version
Key Benefits
• Centralized prompt version management
• Easy A/B testing of prompt variations
• Collaborative prompt optimization
Potential Improvements
• Automated prompt suggestion system
• Context-aware prompt selection
• Integration with external knowledge bases
Business Value
.svg)
Efficiency Gains
Reduces prompt development time by 50% through reusable templates
.svg)
Cost Savings
Minimizes token usage by identifying optimal prompt structures
.svg)
Quality Improvement
Increases accuracy of disjointness detection through systematic prompt refinement