| Property | Value |
|---|---|
| Parameter Count | 7 Billion |
| Context Length | 32K tokens |
| Languages | English, French, Spanish, Portuguese |
| License | TII Falcon-LLM License 2.0 |
| Release Date | December 2024 |
What is Falcon3-7B-Instruct?
Falcon3-7B-Instruct is a state-of-the-art instruction-tuned language model developed by the Technology Innovation Institute (TII). It represents a significant advancement in multilingual AI capabilities, having been pretrained on 14 Teratokens of diverse datasets and fine-tuned on 1.2 million samples of specialized content including STEM, conversations, code, and safety data.
Implementation Details
The model employs a transformer-based causal decoder architecture with 28 decoder blocks. It features advanced technical innovations including Grouped Query Attention (GQA) with 12 query heads and 4 key-value heads, enhanced with a wider head dimension of 256 and high RoPE value of 1000042 for improved long-context understanding.
- Architecture: Transformer-based causal decoder with SwiGLU and RMSNorm
- Vocabulary Size: 131K tokens
- Training Infrastructure: Utilized 1024 H100 GPU chips
- Context Window: Supports up to 32K tokens
Core Capabilities
- Exceptional performance in STEM and mathematical reasoning tasks (31.87% on MATH Lvl-5)
- Strong multilingual support across four major languages
- Advanced reasoning capabilities (37.92% on BBH 3-shot)
- Robust instruction following (76.12% on IFEval)
- Impressive scientific question answering (94.7% on SciQ)
Frequently Asked Questions
Q: What makes this model unique?
Falcon3-7B-Instruct stands out for its exceptional performance in STEM and reasoning tasks, multilingual capabilities, and extensive context window. It achieves state-of-the-art results in various benchmarks while maintaining a relatively compact 7B parameter size.
Q: What are the recommended use cases?
The model excels in scientific and mathematical applications, multilingual content generation, long-form content understanding, and instruction-following tasks. It's particularly well-suited for educational applications, research assistance, and technical documentation generation across multiple languages.