StableCode-Completion-Alpha-3B-4K
| Property | Value |
|---|---|
| Parameter Count | 2.79B |
| Architecture | GPT-NeoX based decoder-only |
| Context Length | 4096 tokens |
| License | Apache 2.0 |
| Training Data | 300B tokens from StarCoder dataset |
What is stablecode-completion-alpha-3b-4k?
StableCode-Completion-Alpha-3B-4K is a specialized code completion model developed by Stability AI. It's a 3 billion parameter decoder-only transformer model designed to handle long-context code completion tasks with a 4k token window. The model has been trained on a diverse set of programming languages from the StarCoder dataset, focusing on languages that top the Stack Overflow developer survey.
Implementation Details
The model architecture features 32 layers with 32 attention heads and a hidden size of 2560. It employs parallel attention and MLP residuals with a single input LayerNorm, complemented by Rotary Position Embeddings for enhanced position awareness. The model utilizes mixed-precision BF16 training and is optimized using AdamW.
- Trained on 300B tokens from diverse programming languages
- Implements Flash Attention-2 for improved efficiency
- Uses StarCoder tokenizer with 49k vocabulary size
- Trained with 2D parallelism and ZeRO-1 optimization
Core Capabilities
- Single/multiline code completion
- Long context understanding (up to 4096 tokens)
- 17.68% pass@1 on HumanEval benchmark
- 27.01% pass@10 performance
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its combination of a substantial parameter count (3B) and an extended context window of 4k tokens, making it particularly effective for understanding and completing longer code segments. The implementation of Flash Attention-2 and Rotary Position Embeddings also contributes to its efficient processing of long sequences.
Q: What are the recommended use cases?
The model is specifically designed for code completion tasks in popular programming languages. It's best suited for developers looking for AI assistance in code generation, with particular strength in handling longer context windows. It's recommended to use it with the HuggingFace VSCode extension for optimal results and proper code attribution.
