Published
Oct 4, 2024
Updated
Oct 4, 2024

Do Avatars Make AI More Aggressive? The Surprising Truth About Visual Personas

Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas
By
Seungjong Sun|Eungu Lee|Seo Yeon Baek|Seunghyun Hwang|Wonbyung Lee|Dongyan Nan|Bernard J. Jansen|Jang Hyun Kim

Summary

Can what an AI “sees” change how it behaves? That's the fascinating question explored by new research examining how visual personas impact the behavior of large language models (LLMs). Researchers gave LLMs fictional avatar images as visual personas, and then observed their negotiation strategies in a simulated game. The results revealed that LLMs with more aggressive-looking avatars tended to make bolder, more self-serving offers in the game. Interestingly, these AI agents also seemed to assess the aggressiveness of their opponents' avatars, adjusting their tactics accordingly. When faced with a less aggressive-looking opponent, they took a more dominant stance. Conversely, they were more likely to concede when facing opponents with intimidating avatars. This suggests that, similar to humans, visual cues play a significant role in how LLMs perceive social dynamics and make decisions. The implications are far-reaching. As LLMs become more integrated into our lives, understanding how visual information shapes their interactions is crucial for responsible development. One model, GPT-4o, demonstrated a greater ability to adapt its behavior based on the perceived aggressiveness of its own and its opponent's avatar, exhibiting a sort of “kiss up, kick down” dynamic. This raises important ethical questions: if AI can be influenced by visual cues, how do we prevent biased or harmful behaviors from emerging? While this research focused on aggressiveness, it opens doors to explore how a broader range of visual traits could shape LLM behavior. Imagine AIs trained with personas that reflect empathy, helpfulness, or even humor—could this lead to more human-like and positive interactions? There’s much more to uncover, but this research provides a compelling glimpse into the complex interplay between visual perception and AI behavior, highlighting both the potential and the challenges that lie ahead.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team. Get started for free.

Question & Answers

How did researchers implement visual personas to study LLM behavior in negotiation games?
The researchers used a controlled experimental setup where LLMs were assigned fictional avatar images as visual personas during negotiation simulations. The implementation involved: 1) Creating a dataset of avatars with varying levels of perceived aggressiveness, 2) Integrating these avatars into the LLM's context window during negotiations, and 3) Measuring behavioral outcomes through negotiation offers and responses. For example, when GPT-4o was given an aggressive avatar, it made more assertive initial offers and showed less willingness to compromise, similar to how a human might adopt a more dominant stance when projecting a tough image.
How can AI visual personas improve human-AI interactions in everyday applications?
AI visual personas can enhance human-AI interactions by making digital assistants more relatable and context-appropriate. These visual representations help users better understand and predict AI behavior, similar to how we use visual cues in human interactions. Benefits include more natural communication, increased user trust, and better engagement. For instance, a customer service AI might use a friendly, professional avatar to create a more comfortable experience, while an AI fitness coach could use a more energetic persona to better motivate users. This technology is particularly valuable in education, healthcare, and customer service applications.
What role do visual cues play in AI decision-making systems?
Visual cues significantly influence AI decision-making by providing additional context and social information that shapes behavioral responses. These cues help AI systems better understand and adapt to social dynamics, similar to human social intelligence. Key benefits include more nuanced interactions, better emotional intelligence, and improved situational awareness. In practical applications, this could mean AI assistants that adjust their communication style based on visual context - being more formal in professional settings or more casual in social situations. This advancement is particularly relevant for virtual assistants, social robots, and AI-driven customer service platforms.

PromptLayer Features

  1. A/B Testing
  2. Testing different avatar-prompt combinations to evaluate their impact on LLM behavior and negotiation outcomes
Implementation Details
Create test sets with varied avatar-prompt pairs, track response variations, measure aggressiveness metrics
Key Benefits
• Systematic evaluation of visual persona effects • Quantifiable behavior change measurements • Reproducible testing framework
Potential Improvements
• Automated visual trait scoring • Cross-model comparison capabilities • Integrated bias detection metrics
Business Value
Efficiency Gains
50% faster evaluation of persona-based prompt variations
Cost Savings
Reduced testing costs through automated comparison workflows
Quality Improvement
More consistent and unbiased AI interactions across different contexts
  1. Version Control
  2. Managing and tracking different versions of prompts with varying avatar configurations and negotiation strategies
Implementation Details
Create versioned prompt templates with avatar metadata, track behavioral outcomes, maintain history
Key Benefits
• Traceable evolution of persona-based prompts • Easy rollback of problematic configurations • Collaborative prompt optimization
Potential Improvements
• Visual persona metadata tagging • Behavioral impact tracking • Automated version recommendations
Business Value
Efficiency Gains
40% faster prompt iteration cycles
Cost Savings
Reduced development overhead through reusable prompt templates
Quality Improvement
Better control over AI behavior across different versions

The first platform built for prompt engineering