Published
Nov 25, 2024
Updated
Nov 25, 2024

Automating Assertions: AI-Powered Code Validation

ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code
By
Mohammad Jalili Torkamani|Abhinav Sharma|Nikita Mehrotra|Rahul Purandare

Summary

Writing good assertions is crucial for robust software, but it’s often a tedious, manual task. What if AI could automate it? Researchers have developed Assertify, a tool that uses large language models (LLMs) to automatically generate assertions for production code. Think of it like a supercharged spellchecker, not just for grammar, but for the logic within your code. Assertify analyzes your code, understands its intended behavior, and then generates assertions that check for unexpected conditions during runtime. This helps developers catch bugs early, improve code comprehension, and ensure their code does what it’s supposed to do. The research explored several prompt engineering techniques to give the LLM the right context, from basic method signatures to sophisticated few-shot learning with similar code examples. The most effective approach involved showing the LLM examples of how assertions are used in similar code, effectively teaching it to reason about code behavior. Results showed that Assertify could generate syntactically correct assertions with high accuracy, exceeding 97% in some cases, and semantically correct assertions with accuracies reaching 83.5%. While the generated assertions aren’t perfect, they demonstrate the potential of LLMs to automate a critical aspect of software development. Future work focuses on refining the accuracy of generated assertions, supporting more programming languages, and exploring open-source LLMs. The potential impact is significant, freeing developers from tedious manual work and enabling them to build more robust, reliable software.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How does Assertify's prompt engineering technique work to generate accurate code assertions?
Assertify uses few-shot learning by showing the LLM examples of assertions in similar code contexts. The process involves three key steps: 1) Analyzing the target code's method signatures and structure, 2) Providing the LLM with carefully selected examples of similar code with proper assertions, and 3) Using this context to generate new, contextually appropriate assertions. For example, if working with a method handling user authentication, Assertify would show the LLM examples of other authentication-related code with assertions checking for null users, invalid credentials, or session timeouts. This approach achieved up to 97% syntactic accuracy and 83.5% semantic accuracy in assertion generation.
What are the benefits of automated code validation for software development?
Automated code validation streamlines the software development process by automatically checking code quality and correctness. Key benefits include: faster bug detection during development, reduced manual testing effort, and improved code reliability. For example, developers working on a banking application can focus on implementing features while automated validation ensures transaction processing meets security and accuracy requirements. This automation is particularly valuable for large teams working on complex projects, where manual validation would be time-consuming and error-prone. It also helps maintain consistent code quality standards across the entire development lifecycle.
How is AI transforming software testing and quality assurance?
AI is revolutionizing software testing by introducing intelligent automation and predictive analysis capabilities. It helps identify potential bugs before they reach production, generates test cases automatically, and adapts to new code changes without manual intervention. For instance, AI can analyze patterns in historical bug reports to predict vulnerable areas in new code changes. This transformation makes testing more efficient, reduces human error, and allows development teams to deliver higher quality software faster. The technology is particularly valuable for continuous integration/continuous deployment (CI/CD) pipelines, where rapid testing and validation are essential.

PromptLayer Features

  1. Testing & Evaluation
  2. Assertify's evaluation of assertion accuracy aligns with PromptLayer's testing capabilities for measuring prompt performance
Implementation Details
1. Create test suites with known good assertions, 2. Use batch testing to evaluate LLM outputs, 3. Track accuracy metrics across prompt versions
Key Benefits
• Automated accuracy measurement of generated assertions • Comparison tracking across different prompt versions • Systematic evaluation of few-shot learning effectiveness
Potential Improvements
• Integration with code testing frameworks • Custom metrics for assertion quality • Automated regression testing pipelines
Business Value
Efficiency Gains
Reduces manual testing effort by 70-80%
Cost Savings
Cuts QA costs by automating assertion verification
Quality Improvement
Ensures consistent assertion quality across codebase
  1. Prompt Management
  2. Research's few-shot learning approach requires careful prompt versioning and example management
Implementation Details
1. Version control different few-shot examples, 2. Create template prompts for different code contexts, 3. Track prompt performance metrics
Key Benefits
• Systematic organization of few-shot examples • Easy modification of prompt strategies • Performance tracking across prompt versions
Potential Improvements
• Dynamic example selection based on code context • Automated prompt optimization • Integration with code repositories
Business Value
Efficiency Gains
Reduces prompt engineering time by 50%
Cost Savings
Optimizes LLM usage through better prompts
Quality Improvement
Increases assertion generation accuracy through better prompt management

The first platform built for prompt engineering