| Property | Value |
|---|---|
| Parameter Count | 3.09B |
| Architecture | Transformers with RoPE, SwiGLU, RMSNorm |
| Context Length | 32,768 tokens |
| License | Qwen Research |
| Paper | Technical Report |
What is Qwen2.5-Coder-3B?
Qwen2.5-Coder-3B is part of the latest series of Code-Specific Qwen large language models, specifically designed for code-related tasks. This 3B parameter model represents a significant advancement in code generation and reasoning capabilities, built upon the strong foundation of Qwen2.5.
Implementation Details
The model features a sophisticated architecture utilizing transformers with RoPE, SwiGLU, and RMSNorm. It implements 36 layers with 16 attention heads for queries and 2 for key-values (GQA), supporting an impressive context length of 32,768 tokens. The model has been trained on 5.5 trillion tokens including source code, text-code grounding, and synthetic data.
- Non-Embedding Parameters: 2.77B
- Attention Structure: 16 heads for Q, 2 for KV
- Advanced Features: Attention QKV bias and tied word embeddings
Core Capabilities
- Enhanced code generation and reasoning
- Improved code fixing functionality
- Strong mathematical reasoning abilities
- Suitable for Code Agent applications
Frequently Asked Questions
Q: What makes this model unique?
The model combines advanced architecture with specialized code training, making it particularly effective for code-related tasks while maintaining strong general capabilities. Its 32K context length and efficient attention mechanism make it particularly suitable for handling large code blocks.
Q: What are the recommended use cases?
While the model excels at code generation, reasoning, and fixing, it's not recommended for direct conversations. Instead, it's best suited for post-training applications, SFT, RLHF, or continued pretraining tasks.