| Property | Value |
|---|---|
| Parameter Count | 15 Billion |
| License | BigCode OpenRAIL-M |
| Paper | arXiv:2410.24198 |
| Training Data | self-oss-instruct-sc2-exec-filter-50k |
| Model Type | Instruction-tuned Code Generation |
What is StarCoder2-15B-Instruct-v0.1?
StarCoder2-15B-Instruct-v0.1 represents a groundbreaking achievement in code language models as the first entirely self-aligned code LLM trained with a fully permissive and transparent pipeline. Built on the foundation of StarCoder2-15B, this model has been fine-tuned using self-generated instruction-response pairs, achieving impressive performance across multiple code generation benchmarks.
Implementation Details
The model utilizes the Transformers architecture and is trained using Adafactor optimizer with a learning rate of 1e-5. Training was conducted over 4 epochs with a batch size of 64 and sequence length of 1280. The model employs BF16 precision and includes a linear learning rate scheduler with 0.05 warmup ratio.
- Achieves 72.6% pass@1 on HumanEval
- 75.2% pass@1 on MBPP benchmark
- 40.6% pass@1 on DS-1000
- Optimized for single-turn coding instructions
Core Capabilities
- Python code generation with type hints
- Custom sorting and algorithm implementation
- Test output prediction (29.8% pass@1)
- Code execution tasks (28.1% pass@1)
- Self-repair capabilities (20.9% pass@1)
Frequently Asked Questions
Q: What makes this model unique?
This is the first code LLM that's entirely self-aligned, meaning it generates its own instruction-response pairs without relying on human annotations or proprietary LLM data. This makes it fully transparent and permissive for use.
Q: What are the recommended use cases?
The model excels at Python code generation tasks, particularly those that can be verified through execution. It's best suited for single-turn coding instructions and performs optimally when provided with clear, specific programming tasks.