Large language models (LLMs) are increasingly powerful, but they're also vulnerable to adversarial attacks, or 'red teaming,' that can expose their weaknesses and biases. Traditional red teaming methods often struggle to balance the need for diverse attacks with a high success rate. This research introduces a new two-step approach that leverages automatically generated rewards and multi-step reinforcement learning to create a more effective and diverse AI red teamer. The first step involves generating a wide array of attack goals, drawing from sources like existing attack datasets and few-shot prompting of LLMs. These goals aren't designed to be immediately effective attacks, but rather serve as diverse starting points. For instance, instead of simply aiming to elicit unsafe content, the goals might be broken down into more specific objectives like "instructions to hijack a car" or "guidance for how to launder money." Each goal is then paired with a specific criterion for success, forming the basis for a targeted, rule-based reward. The second step involves training an attacker LLM using reinforcement learning (RL). This attacker receives a goal as input and attempts to generate a prompt that achieves that goal. Crucially, the attacker is rewarded not just for success, but also for generating attacks that are different from previous attempts. This multi-step RL process, combined with a novel diversity reward focused on the style and tactics of the attack, encourages the attacker to explore a wider range of strategies. For example, while two prompts might both aim to elicit unsafe content, one might use satire while the other uses direct questioning. This focus on stylistic diversity helps uncover a broader spectrum of vulnerabilities. The researchers tested their method on two tasks: prompting the model to follow instructions from irrelevant third-party inputs (indirect prompt injection) and eliciting unsafe responses (safety jailbreaking). The results show that this new approach effectively balances diversity and effectiveness. In the case of indirect prompt injections, the method generated a wider range of successful attacks than traditional methods. For safety jailbreaking, the approach significantly improved attack diversity while maintaining a high success rate. While challenges remain, such as the difficulty of measuring diversity and the potential for 'reward hacking,' this research provides valuable new techniques for building more robust and secure AI systems.