| Property | Value |
|---|---|
| Parameter Count | 24.2B |
| Tensor Type | BF16 |
| License | CC-BY-NC-4.0 |
| Architecture | Mixture of Experts (MoE) |
What is mixtralnt-4x7b-test?
Mixtraln't 4x7B is an experimental Mixture of Experts (MoE) model that takes an innovative approach to model architecture. Instead of training a MoE from scratch, it combines multiple pre-trained Mistral-7B models into a single architecture, metaphorically described as putting them in a "clown car." This unique approach explores the possibility of leveraging existing pre-trained models in a MoE framework.
Implementation Details
The model incorporates components from five different Mistral-based models: MetaMath-Cybertron-Starling, Noromaid-7b, Mistral-Trismegistus-7B, MetaMath-Mistral-7B, and Dans-AdventurousWinds-Mk2-7b. The implementation uses a custom hack for MoE gate population to potentially utilize all experts effectively.
- 24.2B total parameters across combined models
- BF16 precision for efficient computation
- Custom MoE gate implementation
- Built on Mistral-7B architecture variants
Core Capabilities
- Text generation with coherent outputs
- Flexible prompt format supporting multiple styles
- Potential for diverse expertise from different base models
- Experimental architecture for research purposes
Frequently Asked Questions
Q: What makes this model unique?
This model's uniqueness lies in its experimental approach to creating a MoE system by combining pre-trained models rather than training from scratch. This "clown car" approach to model architecture is both innovative and unconventional.
Q: What are the recommended use cases?
The model is primarily suited for research and experimental purposes, particularly in studying how multiple pre-trained models can work together in a MoE architecture. It supports various prompt formats including alpaca and chatml, making it flexible for different text generation tasks.