StableCode-Completion-Alpha-3B
| Property | Value |
|---|---|
| Parameters | 2.79B |
| Architecture | GPT-NeoX based Decoder-only |
| Context Length | 16,384 tokens |
| License | Apache 2.0 |
| Training Data | StarCoder Dataset |
What is stablecode-completion-alpha-3b?
StableCode-Completion-Alpha-3B is a specialized code completion model developed by Stability AI, designed to assist developers with intelligent code suggestions. Built on a transformer decoder architecture, this model leverages 2.79B parameters to understand and generate code across multiple programming languages. It has been trained on the comprehensive StarCoder dataset, focusing on the most popular programming languages according to the 2023 Stack Overflow developer survey.
Implementation Details
The model incorporates several advanced technical features, including parallel attention and MLP residuals with single input LayerNorm, and Rotary Position Embeddings. It was trained in two stages: initial pre-training with 300B tokens at 4096 context length, followed by fine-tuning with 200B tokens at 16384 context length.
- Hidden Size: 2560
- Number of Layers: 32
- Attention Heads: 32
- Maximum Sequence Length: 16384 tokens
- Training Precision: Mixed BF16
Core Capabilities
- Long-context code completion up to 16k tokens
- Multi-language support for popular programming languages
- Pass@1 accuracy of 20.18% on HumanEval benchmark
- Pass@10 accuracy of 33.75%
- Efficient single/multiline code completion
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive features include its extended context window of 16k tokens, two-stage training process, and optimization for modern programming languages. It combines flash-attention and rotary embedding kernels from FlashAttention-2 for improved performance.
Q: What are the recommended use cases?
The model is specifically designed for code completion tasks, making it ideal for developers seeking intelligent code suggestions. It's recommended to use it with the HuggingFace VSCode extension for optimal results and proper code attribution.
