replit-code-v1-3b
| Property | Value |
|---|---|
| Model Size | 2.7B parameters |
| License | CC BY-SA-4.0 |
| Training Tokens | 525B |
| Supported Languages | 20 programming languages |
What is replit-code-v1-3b?
replit-code-v1-3b is a sophisticated code completion model developed by Replit, Inc. This 2.7B parameter model has been specifically designed for code generation and completion tasks across multiple programming languages. Trained on a carefully curated subset of the Stack Dedup v1.2 dataset, it processes code in 20 different programming languages, including popular ones like Python, JavaScript, Java, and TypeScript.
Implementation Details
The model leverages cutting-edge LLM techniques to deliver optimal performance. It incorporates Flash Attention for efficient training and inference, AliBi positional embeddings for handling variable context lengths, and the LionW optimizer for improved training dynamics. The model was trained using 256 A100-40GB GPUs on the MosaicML platform, processing 175B tokens over 3 epochs for a total of 525B tokens.
- Custom SentencePiece Unigram tokenizer with 32,768 tokens
- Support for 8-bit and 4-bit quantization
- Flexible deployment options with PyTorch integration
- Built-in support for Triton implementation of FlashAttention
Core Capabilities
- Multi-language code completion across 20 programming languages
- Efficient handling of various programming paradigms
- Support for both completion and generation tasks
- Optimized performance with Flash Attention and AliBi embeddings
- Flexible deployment options including quantization support
Frequently Asked Questions
Q: What makes this model unique?
The model's unique strength lies in its comprehensive training across 20 programming languages and its implementation of state-of-the-art techniques like Flash Attention and AliBi embeddings. With 195 tokens per parameter during training, it achieves an efficient parameter-to-performance ratio.
Q: What are the recommended use cases?
The model is primarily designed for code completion and generation tasks. It's particularly suitable for developers looking for an AI assistant in multi-language development environments, and for businesses seeking to integrate code generation capabilities into their tools.
