Published
Jun 25, 2024
Updated
Oct 16, 2024

Can AI Be Persuasive? Measuring the Rhetoric of LLMs

Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language
By
Amalie Brogaard Pauli|Isabelle Augenstein|Ira Assent

Summary

Have you ever felt subtly swayed by an ad or a piece of writing? That's the power of persuasive language at play. Now, imagine that same power wielded by artificial intelligence. A new research paper explores the fascinating and potentially concerning ability of Large Language Models (LLMs) to generate persuasive text. Researchers delved into this by creating pairs of texts—an original and an LLM-generated paraphrase—designed to be more or less persuasive. These pairs were then judged by human annotators who rated the relative persuasiveness of each. This process resulted in a new dataset, PERSUASIVE-PAIRS, offering a rich resource for understanding how LLMs craft influential language. To go beyond individual judgments, the researchers trained a model to predict persuasiveness scores, creating a benchmark for evaluating different LLMs and their settings. Intriguingly, they found that LLMs, when simply asked to paraphrase without explicit instructions about persuasion, often toned down the persuasiveness of already persuasive text. However, the 'persona' given to an LLM through its system prompt plays a significant role. For example, instructing an LLM to write like a 'tabloid journalist' resulted in significantly more persuasive text compared to a 'scientific journalist.' Similarly, the political leaning assigned to an LLM also affected the persuasiveness of its output. This research highlights the surprising power of LLMs to not only understand but also generate language that influences us. While it opens doors for positive applications, it also raises important ethical questions. As LLMs become more integrated into our lives, understanding how they shape our beliefs and opinions becomes crucial. Future research might explore how different cultures perceive persuasive language and examine the specific linguistic techniques LLMs employ to sway their audience.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How did researchers measure and evaluate the persuasiveness of LLM-generated text?
The researchers employed a systematic two-step approach. First, they created PERSUASIVE-PAIRS, a dataset containing pairs of original and LLM-generated paraphrased texts. Human annotators rated the relative persuasiveness of each pair. Subsequently, they developed a model trained on these human judgments to predict persuasiveness scores, establishing a benchmark for evaluating different LLMs. This methodology allowed for both qualitative human assessment and quantitative computational analysis. For example, this could be used to analyze marketing copy, where two versions of ad text could be compared for their persuasive potential before deployment.
How can AI-generated content influence consumer behavior?
AI-generated content can influence consumer behavior through sophisticated language patterns and personalized messaging. The technology can analyze successful persuasive techniques and replicate them at scale, creating compelling marketing messages, product descriptions, and advertisements. This capability helps businesses better connect with their target audience through more engaging content. For instance, e-commerce platforms can use AI to generate product descriptions that highlight benefits most appealing to specific customer segments, or email marketing campaigns can be optimized for better conversion rates through AI-crafted subject lines and copy.
What are the potential risks of AI-powered persuasive writing in everyday life?
AI-powered persuasive writing poses several significant risks in daily life. It could be used to create highly targeted and manipulative content, potentially influencing people's decisions without their awareness. This technology might be employed in spreading misinformation, creating deceptive advertising, or manipulating public opinion on social media. For example, AI could generate convincing fake reviews, misleading news articles, or persuasive scam messages. Understanding these risks is crucial for developing digital literacy and implementing appropriate safeguards to protect consumers from manipulation.

PromptLayer Features

  1. A/B Testing
  2. Directly aligns with the paper's methodology of comparing paired texts and different prompt personas for persuasiveness
Implementation Details
Configure A/B tests comparing different system prompts and personas, track persuasiveness metrics, analyze performance variations
Key Benefits
• Systematic comparison of prompt effectiveness • Data-driven optimization of persuasive language • Quantifiable performance metrics
Potential Improvements
• Integrate automated persuasiveness scoring • Add demographic-based testing segments • Implement real-time feedback loops
Business Value
Efficiency Gains
Reduce time spent on manual prompt optimization by 60%
Cost Savings
Lower content generation costs through optimized prompt selection
Quality Improvement
20% increase in content persuasiveness through systematic testing
  1. Prompt Management
  2. Supports tracking and versioning different persona-based prompts and their persuasiveness outcomes
Implementation Details
Create versioned prompt templates for different personas, track performance metrics, maintain prompt history
Key Benefits
• Centralized prompt repository • Version control for different personas • Performance tracking across versions
Potential Improvements
• Add persuasiveness scoring metadata • Implement persona categorization • Create prompt effectiveness rankings
Business Value
Efficiency Gains
30% faster prompt iteration and optimization
Cost Savings
Reduced duplicate prompt development effort
Quality Improvement
Consistent persuasiveness across content through standardized prompts

The first platform built for prompt engineering