Llama3-TenyxChat-70B
| Property | Value |
|---|---|
| Parameter Count | 70.6B |
| License | Meta Llama 3 Community License |
| Base Model | Llama3-70B |
| Training Framework | Direct Preference Optimization (DPO) |
| MT-Bench Score | 8.15 |
What is Llama3-TenyxChat-70B?
Llama3-TenyxChat-70B is a state-of-the-art language model developed by Tenyx Research, fine-tuned from Meta's Llama3-70B using advanced preference optimization techniques. It represents a significant achievement in open-source AI, achieving the highest MT-Bench score (8.15) among publicly available models.
Implementation Details
The model was trained using eight A100 GPUs (80GB) for fifteen hours, implementing Direct Preference Optimization on the UltraFeedback dataset. Tenyx's proprietary approach focuses on mitigating catastrophic forgetting while maintaining high performance across various benchmarks.
- Uses BF16 tensor type for efficient computation
- Compatible with standard Llama3 chat templates
- Evaluated using GPT-4-preview-0125 for benchmarking
- Demonstrates exceptional performance in both single and multi-turn conversations
Core Capabilities
- Achieves 8.15 on MT-Bench, surpassing the base Llama3-70B-Instruct (7.96)
- Scores 79.43 average on Open LLM Leaderboard benchmarks
- Maintains consistent performance in multi-turn dialogues
- Excels in GSM8K (91.21%) and HellaSwag (86.11%) tasks
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive feature is its ability to maintain high performance across multiple conversation turns while avoiding catastrophic forgetting, achieved through Tenyx's specialized fine-tuning approach.
Q: What are the recommended use cases?
The model excels in conversational AI applications, particularly in scenarios requiring sustained dialogue quality across multiple turns. It's suitable for both general-purpose chat applications and specialized tasks in reasoning, STEM, and humanities domains.
