Can AI Master Deception? LLMs Play Among Us
AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game
By
Yizhou Chi|Lingjun Mao|Zineng Tang

https://arxiv.org/abs/2407.16521v2
Summary
Imagine an AI, not just playing a game, but strategizing, deceiving, and collaborating like a human. That's the fascinating premise behind new research exploring how Large Language Models (LLMs) perform in the popular social deduction game, Among Us. Researchers created a text-based version of the game, called AMONGAGENTS, to see if LLMs could grasp the complex rules and social dynamics. Could they successfully deceive as impostors, or effectively collaborate as crewmates to identify the saboteurs? The results reveal some intriguing insights into the capabilities and limitations of today's AI. LLMs showed a surprising ability to understand the game's rules, often making decisions based on observed events. As impostors, they employed deception, strategically using vents and crafting alibis. Crewmates, on the other hand, focused on completing tasks and sharing information, exhibiting collaborative behavior. However, there is a lot of progress needed in social deduction games, as playing and mastering deception and trickery in text-based games is still a developing area for current LLMs. While LLMs demonstrated an understanding of the rules, their strategies weren't always foolproof. Impostors sometimes struggled to maintain their deception, and crewmates occasionally made inaccurate accusations. The study also explored different LLM 'personalities,' such as 'The Strategist,' 'The Skeptic,' or 'The Manipulator,' revealing how these different profiles influenced gameplay. The research highlights the growing potential of LLMs in navigating complex social situations, even within the constraints of a game. While they haven't quite mastered the art of deception, these early experiments hint at a future where AI can truly understand, and perhaps even outsmart, human social dynamics. Future research aims to refine LLM strategies for social deduction games, including more nuanced personalities and improved deception skills. The ultimate goal is to develop AI that can truly engage in complex social interactions, with potential applications far beyond the gaming world.
🍰 Interesting in building your own agents?
PromptLayer provides the tools to manage and monitor prompts with your whole team.
Get started for free.Question & Answers
How did researchers implement different AI personalities in AMONGAGENTS, and what was their technical approach?
Researchers created distinct AI personalities like 'The Strategist,' 'The Skeptic,' and 'The Manipulator' by implementing specific behavior profiles in the LLMs. The technical approach involved: 1) Defining characteristic behaviors and decision-making patterns for each personality type, 2) Training the LLMs to respond according to these predetermined patterns during gameplay, and 3) Implementing rule-based frameworks that guided how each personality would interact with other players and approach game objectives. For example, 'The Strategist' might prioritize logical deduction and evidence-based accusations, while 'The Manipulator' would focus on social engineering tactics to influence other players' decisions.
What are the real-world applications of AI systems that can understand social dynamics?
AI systems that understand social dynamics have numerous practical applications across various sectors. In customer service, they can better interpret customer intentions and emotions to provide more empathetic responses. In education, they can serve as more engaging tutors by adapting to students' social and emotional needs. In healthcare, these systems could assist in mental health support by better understanding patient communication patterns. The technology could also enhance virtual assistants, making them more natural and context-aware in their interactions, ultimately leading to more effective human-AI collaboration in professional and personal settings.
How is AI being used to improve gaming experiences?
AI is revolutionizing gaming experiences through several key innovations. It creates more realistic and adaptive NPCs (Non-Player Characters) that can learn from player behavior and provide more engaging interactions. AI also enables dynamic difficulty adjustment, automatically tailoring game challenges to match player skill levels. In multiplayer games, AI can enhance matchmaking systems by analyzing player patterns and preferences. Additionally, AI is being used to generate unique content, create more immersive storylines, and even help develop more sophisticated game testing methods. These applications make games more engaging, personalized, and enjoyable for players of all skill levels.
.png)
PromptLayer Features
- Testing & Evaluation
- The paper's evaluation of different LLM 'personalities' and their performance in deception/collaboration aligns with systematic prompt testing needs
Implementation Details
Create test suites for different personality prompts, run batch tests across various game scenarios, measure success rates of deception and collaboration
Key Benefits
• Systematic evaluation of personality prompt effectiveness
• Quantifiable measurement of deception success rates
• Reproducible testing across different game scenarios
Potential Improvements
• Add automated personality prompt generation
• Implement real-time performance metrics
• Develop specialized scoring for deception success
Business Value
.svg)
Efficiency Gains
Reduces manual testing time by 70% through automated evaluation pipelines
.svg)
Cost Savings
Minimizes prompt development costs through systematic testing
.svg)
Quality Improvement
Ensures consistent personality behavior across different scenarios
- Analytics
- Workflow Management
- Managing complex game strategies and multiple personality types requires sophisticated prompt orchestration and version tracking
Implementation Details
Create reusable templates for different personalities, implement version control for strategy evolution, establish multi-step game flow orchestration
Key Benefits
• Streamlined management of multiple personality prompts
• Traceable evolution of strategy improvements
• Consistent game flow across different scenarios
Potential Improvements
• Add dynamic personality switching based on game context
• Implement strategy optimization workflows
• Create automated performance feedback loops
Business Value
.svg)
Efficiency Gains
Reduces prompt management overhead by 50% through templating
.svg)
Cost Savings
Decreases development time through reusable components
.svg)
Quality Improvement
Ensures consistent strategy implementation across different game instances