Published
Nov 23, 2024
Updated
Nov 23, 2024

Can AI Write Bug-Free Hardware?

Automatic High-quality Verilog Assertion Generation through Subtask-Focused Fine-Tuned LLMs and Iterative Prompting
By
Mohammad Shahidzadeh|Behnam Ghavami|Steve Wilton|Lesley Shannon

Summary

Ensuring that computer hardware behaves exactly as intended is a complex process. Formal Property Verification (FPV) uses precise statements, called assertions, to rigorously check hardware designs against their specifications. However, writing these assertions is a tedious, manual task, prone to human error. Could AI automate this crucial step? Researchers explored this question in "Automatic High-quality Verilog Assertion Generation through Subtask-Focused Fine-Tuned LLMs and Iterative Prompting." They developed AssertCraft, a tool that leverages Large Language Models (LLMs) like those powering ChatGPT to generate assertions automatically from hardware design documents. Because LLMs struggle with the multi-step reasoning required for this task, the researchers introduced a clever two-stage approach. First, they "fine-tuned" an LLM on a dataset of existing assertions and comments, training it to understand the relationship between human-language descriptions and formal assertions. Second, they implemented an "iterative prompting" method where a custom compiler provides feedback to the LLM, enabling it to correct errors in its generated assertions. This back-and-forth refinement process significantly improves the quality of the generated assertions. The results are promising. AssertCraft achieved a remarkable 7.3-fold increase in functionally correct assertions compared to a standard LLM. Furthermore, it demonstrated near-perfect coverage in many test cases, meaning it successfully generated assertions that checked most aspects of the hardware’s behavior. This research shows the potential of AI to automate complex hardware verification tasks, leading to faster development cycles and more reliable hardware. While challenges remain in refining the accuracy and coverage of LLM-generated assertions, this work points toward a future where AI plays a critical role in ensuring the bug-free operation of the chips powering our world.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does AssertCraft's two-stage approach work to generate hardware assertions?
AssertCraft uses a two-stage approach combining fine-tuning and iterative prompting. First, it fine-tunes an LLM on existing assertions and comments to understand the relationship between natural language and formal assertions. Then, it implements an iterative feedback loop where a compiler reviews generated assertions and provides feedback to the LLM for improvement. This process achieved a 7.3x increase in functionally correct assertions compared to standard LLMs. For example, when verifying a memory controller, AssertCraft could generate assertions checking data integrity, timing constraints, and protocol compliance while refining these checks based on compiler feedback until they're functionally correct.
What are the benefits of AI-powered hardware verification for everyday technology?
AI-powered hardware verification makes our electronic devices more reliable and gets them to market faster. By automating the complex process of checking hardware designs, AI helps catch potential bugs before chips are manufactured, reducing the chance of problems in smartphones, laptops, and other devices we use daily. This means fewer product recalls, better performance, and quicker releases of new technology. For example, when a new smartphone processor is being developed, AI verification tools can help ensure it works correctly under all conditions, leading to better battery life and fewer crashes.
How is AI changing the future of computer chip design?
AI is revolutionizing computer chip design by automating complex verification processes and reducing human error. Traditional chip design requires extensive manual testing and verification, but AI tools can now automatically generate and verify design specifications, potentially cutting development time from months to weeks. This advancement means faster innovation cycles, more reliable electronics, and potentially lower costs for consumers. In practical terms, this could lead to more frequent releases of improved processors for our devices, better energy efficiency, and more innovative features in future electronics.

PromptLayer Features

  1. Workflow Management
  2. The paper's two-stage approach with iterative refinement closely mirrors multi-step prompt orchestration needs
Implementation Details
1. Create template for initial assertion generation 2. Build feedback loop with compiler results 3. Configure iteration logic based on quality metrics
Key Benefits
• Reproducible multi-stage prompting flows • Structured handling of compiler feedback • Version tracking across refinement iterations
Potential Improvements
• Add automated quality gates between stages • Implement parallel refinement paths • Create specialized templates for different hardware components
Business Value
Efficiency Gains
Reduced manual oversight needed for complex prompt chains
Cost Savings
Fewer LLM calls through optimized iteration paths
Quality Improvement
More consistent and traceable assertion generation process
  1. Testing & Evaluation
  2. The research's focus on assertion quality and coverage metrics aligns with systematic prompt testing needs
Implementation Details
1. Define assertion quality metrics 2. Create test suites for different hardware scenarios 3. Implement automated scoring pipeline
Key Benefits
• Systematic quality assessment • Comparable performance metrics • Regression prevention
Potential Improvements
• Expand test coverage metrics • Add automated regression testing • Implement peer comparison benchmarks
Business Value
Efficiency Gains
Faster identification of prompt performance issues
Cost Savings
Reduced debugging time through systematic testing
Quality Improvement
Higher reliability in generated assertions

The first platform built for prompt engineering