llama-7b-hf-transformers-4.29
| Property | Value |
|---|---|
| Model Size | 7B parameters |
| Architecture | Transformer-based LLM |
| License | Non-commercial bespoke license |
| Training Data | CCNet (67%), C4 (15%), GitHub (4.5%), Wikipedia (4.5%), Books (4.5%), ArXiv (2.5%), Stack Exchange (2%) |
What is llama-7b-hf-transformers-4.29?
This is a conversion of Meta AI's LLaMA 7B model optimized for the Transformers 4.29 framework. It represents a state-of-the-art language model featuring 4096 dimensional embeddings, 32 attention heads, and 32 layers. The model was trained between December 2022 and February 2023 by Meta's FAIR team.
Implementation Details
The model architecture utilizes advanced transformer technology with specific hyperparameters including a learning rate of 3.0E-04 and a batch size of 4M. It was trained on 1T tokens across multiple languages, though predominantly English content.
- Dimension size: 4096
- Number of attention heads: 32
- Number of layers: 32
- Training corpus: Multi-lingual with focus on English
Core Capabilities
- Common sense reasoning (76.5% on BoolQ)
- Reading comprehension
- Natural language understanding
- Question answering
- Multi-lingual text processing (20 languages supported)
Frequently Asked Questions
Q: What makes this model unique?
This model represents a highly efficient implementation of the LLaMA architecture, optimized specifically for the Transformers 4.29 framework. It provides a balance between model size and performance, achieving strong results across various benchmarks while maintaining a relatively modest 7B parameter count.
Q: What are the recommended use cases?
The model is primarily intended for research purposes in natural language processing, including exploring applications in question answering, natural language understanding, and reading comprehension. It should not be used in production environments without further fine-tuning and risk evaluation.
