| Property | Value |
|---|---|
| Parameter Count | 3.32B |
| License | Apache 2.0 |
| Context Length | 4096 tokens |
| Training Data | 1T tokens |
| Architecture | Causal Language Model |
What is replit-code-1.5?
Replit Code V-1.5 is a specialized code completion model trained by Replit using MosaicML's platform. This 3.32B parameter model is designed specifically for code generation tasks, trained on a diverse dataset of 30 programming languages from the Stack Dedup dataset and RedPajama's StackExchange data.
Implementation Details
The model utilizes a custom-trained GPTNeoX tokenizer with an optimized vocabulary of 32,768 tokens, achieving improved compression while maintaining high coverage. Training was conducted on 128 H100-80GB GPUs using MosaicML's LLM Foundry and Composer framework.
- Trained in bfloat16 precision
- Supports 30 programming languages including Python, JavaScript, Java, C++, and more
- Implements Triton Flash Attention for improved performance
- Context window of 4096 tokens
Core Capabilities
- Advanced code completion across multiple programming languages
- Natural language processing for development-related content
- Efficient token compression with optimized vocabulary
- Support for both CPU and GPU inference
Frequently Asked Questions
Q: What makes this model unique?
The model's specialized training on permissively licensed code and its optimized vocabulary make it particularly effective for code completion tasks while maintaining efficient token compression.
Q: What are the recommended use cases?
The model is best suited for code completion tasks, development assistance, and as a foundation for application-specific fine-tuning in commercial and non-commercial settings. It's particularly effective when used with appropriate temperature and repetition penalty settings for optimal performance.