Published
Jul 18, 2024
Updated
Aug 16, 2024

Can AI Help Us Build Safer Software? CoDefeater and the Future of Assurance Cases

CoDefeater: Using LLMs To Find Defeaters in Assurance Cases
By
Usman Gohar|Michael C. Hunter|Robyn R. Lutz|Myra B. Cohen

Summary

Building safety-critical systems, like self-driving cars or medical devices, is incredibly complex. How do we *really* know they'll work as intended, every single time? One method is building an "assurance case"—a structured argument supported by evidence that demonstrates the system's safety. Think of it like a rigorous proof, meticulously demonstrating why a plane won't fall out of the sky or a nuclear reactor won't melt down. But assurance cases can be vulnerable—gaps in reasoning or missing "edge cases" can lead to disastrous failures. Researchers are constantly searching for these vulnerabilities, known as *defeaters*. Imagine trying to poke holes in the most complex, airtight argument ever assembled. It's challenging, requires creativity, and is prone to human biases like overlooking inconvenient truths. That’s where a new idea called *CoDefeater* comes in. Researchers are exploring whether Large Language Models (LLMs), the technology behind chatbots like ChatGPT, can help automate the hunt for defeaters. LLMs can process huge amounts of information and generate novel arguments, making them potentially great at spotting hidden flaws. Initial research using ChatGPT on two real-world systems—the Large Hadron Collider and a small drone battery system—has shown promising results. The LLM successfully identified many known defeaters, and even generated new, previously undiscovered ones! For the drone battery, the LLM suggested some very practical defeaters, such as unexpected power drain due to component failure, emergency flight extensions, and the surprisingly frequent bird attacks which force drones to take evasive maneuvers. However, LLMs aren't perfect. They sometimes struggle with implicit assumptions in assurance cases, where defeaters challenge not the stated logic, but the underlying premises on which the logic rests. For example, the LLM might not challenge an assumption that a certain frequency signal is the correct one to monitor, whereas a human analyst might identify that using this frequency might mask dangerous interference on another nearby frequency. The future of CoDefeater lies in refining how we interact with these LLMs through better "prompt engineering" (basically, giving the LLM clearer instructions) and incorporating external knowledge bases to help LLMs reason more effectively. This research is just the beginning, but it opens exciting doors. Imagine a world where AI assists safety engineers in building ironclad arguments for the dependability of critical systems. CoDefeater suggests this could become a reality, paving the way for a future with safer and more reliable technology.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does CoDefeater use LLMs to identify vulnerabilities in safety-critical systems?
CoDefeater leverages Large Language Models to analyze assurance cases and identify potential defeaters (vulnerabilities) in safety-critical systems. The system processes comprehensive safety arguments and generates potential failure scenarios or logical gaps that might compromise system safety. For example, when analyzing a drone battery system, the LLM identified both known issues (component failures) and novel defeaters (bird attacks requiring evasive maneuvers). The process involves prompt engineering to guide the LLM's analysis and integration with external knowledge bases to enhance reasoning capabilities. However, the system currently has limitations in identifying implicit assumptions in assurance cases, requiring human oversight for comprehensive safety analysis.
How is AI improving safety testing in modern technology?
AI is revolutionizing safety testing by automating and enhancing the verification process of complex systems. It can quickly analyze vast amounts of data and scenarios that would take humans months or years to review. Benefits include faster detection of potential safety issues, more comprehensive testing coverage, and the ability to identify unexpected failure modes. This technology is being applied across industries, from automotive safety testing for self-driving cars to medical device verification and aerospace systems. For everyday consumers, this means safer products and increased confidence in critical technologies we rely on daily.
What are assurance cases and why are they important for public safety?
Assurance cases are structured arguments supported by evidence that demonstrate a system's safety and reliability. Think of them as detailed safety proofs that show why critical systems, like medical devices or aircraft, won't fail during operation. They're essential because they help prevent accidents and protect public safety by systematically identifying and addressing potential risks. In everyday life, assurance cases impact everything from the elevators we ride to the pacemakers that keep hearts beating. They provide a framework for engineers and safety experts to ensure that complex systems meet strict safety requirements before being deployed in the real world.

PromptLayer Features

  1. Testing & Evaluation
  2. CoDefeater's need to validate LLM-generated defeaters against known vulnerabilities aligns with PromptLayer's testing capabilities
Implementation Details
1. Create test suite of known defeaters, 2. Run batch tests comparing LLM outputs against known cases, 3. Track accuracy metrics over time
Key Benefits
• Systematic validation of LLM-generated defeaters • Historical performance tracking across model versions • Automated regression testing for quality assurance
Potential Improvements
• Integration with domain-specific validation tools • Custom scoring metrics for defeater quality • Automated test case generation
Business Value
Efficiency Gains
Reduces manual validation effort by 70%
Cost Savings
Cuts testing time and resources by half
Quality Improvement
Ensures consistent defeater quality across different systems
  1. Prompt Management
  2. The paper's emphasis on refining prompt engineering for better LLM reasoning maps directly to prompt versioning and optimization
Implementation Details
1. Create template prompts for different system types, 2. Version control prompt iterations, 3. Track effectiveness of different prompt strategies
Key Benefits
• Systematic prompt improvement process • Reusable prompt templates across projects • Clear audit trail of prompt evolution
Potential Improvements
• Domain-specific prompt libraries • Automated prompt optimization • Context-aware prompt selection
Business Value
Efficiency Gains
30% faster prompt development cycle
Cost Savings
Reduced iteration costs through reusable components
Quality Improvement
More consistent and effective LLM outputs

The first platform built for prompt engineering