Imagine a world where artificial intelligence helps keep medications safe. That's the promise of pharmacovigilance, the science of monitoring drug safety. Large language models (LLMs), a type of AI, are being explored for their ability to analyze vast amounts of data, potentially identifying safety signals faster and more efficiently than ever before. But what happens when these powerful AI systems make mistakes? Researchers are tackling this challenge head-on by developing "guardrails" for LLMs in drug safety. These safeguards prevent AI from "hallucinating" or fabricating information, which could have serious consequences for patient health. The study focused on a real-world application: translating Japanese adverse event reports into English for analysis. Researchers found that while LLMs showed promise in handling this complex task, errors did occur. These ranged from incorrect drug names and dosages to misinterpretations of critical adverse events. To address these issues, the team implemented a series of hard and soft guardrails. Hard guardrails act as strict rules, preventing certain errors entirely, such as flagging an AI-generated drug name not found in the original report. Soft guardrails, on the other hand, quantify the AI's uncertainty, highlighting sections of text that might be inaccurate and require human review. The results are encouraging. Hard guardrails successfully prevented all instances of "hallucinated" drug names, a critical safety improvement. Soft guardrails also proved valuable by directing human reviewers to potential errors, enhancing the overall accuracy of the system. While this research focused on translating Japanese reports, the framework has broad implications for AI in safety-critical medical settings. The development and deployment of such guardrails will be crucial in ensuring that AI systems can be safely and effectively integrated into drug safety monitoring, ultimately contributing to safer patient care. Future research will likely focus on expanding and refining these guardrails to tackle a wider range of potential errors and enhance the human-AI collaboration in pharmacovigilance.