Large language models (LLMs) are revolutionizing coding, but can they be trusted with security? New research reveals how LLMs struggle to generate robust code for critical security tasks like detecting attacks. This isn't surprising—LLMs learn from vast datasets, but they don't truly *understand* the nuances of security exploits. Imagine teaching someone to spot counterfeit money just by showing them pictures. They might learn some patterns, but they could easily be fooled by a clever forgery. LLMs face a similar challenge. They can identify known attack patterns, but they lack the deeper reasoning skills to catch sophisticated or novel threats. This research explores how to make LLMs better security guards by adding two key ingredients: external knowledge and self-assessment. First, researchers used a technique called Retrieval Augmented Generation (RAG) to give LLMs access to up-to-date information about attack strategies, similar to giving our money-spotter a guidebook on counterfeiting techniques. Second, they introduced a clever self-ranking mechanism inspired by the concept of 'self-consistency.' This allows the LLM to generate multiple solutions for the same problem and then evaluate which one performs best, like having our trainee compare their analysis against a set of known counterfeits. The results are promising. By integrating these two techniques, the researchers significantly improved the accuracy of LLM-generated code for detecting two common web attacks: cross-site scripting (XSS) and SQL injection (SQLi). The improvement was particularly dramatic for XSS detection, with accuracy boosting by up to 71 percentage points. Importantly, these augmented LLMs achieved near state-of-the-art performance, rivaling specialized machine learning models trained specifically for these security tasks. Furthermore, the research suggests that optimal LLM configurations can be transferred between different security tasks, opening up exciting possibilities for creating more general-purpose AI security tools. While challenges remain, this work highlights the potential of LLMs to play a significant role in enhancing software security, provided they're equipped with the right knowledge and self-evaluation skills.