Replit-v2-CodeInstruct-3B
| Property | Value |
|---|---|
| Base Model | replit/replit-code-v1-3b |
| License | cc-by-sa-4.0 |
| Training Time | 1 hour on 2x A100 80GB GPUs |
| Sequence Length | 2000 tokens |
What is Replit-v2-CodeInstruct-3B?
Replit-v2-CodeInstruct-3B is an advanced code instruction model that builds upon the Replit Code v1 3B base model. It's specifically fine-tuned on a combination of CodeAlpaca and GPTeacher Code-Instruct datasets, comprising approximately 25,000 code instruction/response pairs. This version represents a significant improvement over its predecessor, featuring an extended sequence length of 2000 tokens for enhanced context understanding.
Implementation Details
The model utilizes bfloat16 precision and requires specific implementation parameters for optimal performance. It's designed to work with PyTorch and requires trust_remote_code=True for proper functionality. The model supports various programming languages including Python, JavaScript, Java, TypeScript, PHP, SQL, Rust, and many others.
- Training utilized gradient accumulation steps of 8
- Learning rate of 1e-5 with 3% warmup ratio
- Trained for 3 epochs with specialized saving strategies
- Supports both standard and input-augmented instruction formats
Core Capabilities
- Multi-language code generation and understanding
- Extended context window of 2000 tokens
- Instruction-following capabilities for code-related tasks
- Support for both simple and complex programming queries
- Efficient performance with optimized sampling parameters
Frequently Asked Questions
Q: What makes this model unique?
The model's uniqueness lies in its combination of CodeAlpaca and GPTeacher instruction tuning, along with its extended 2000-token sequence length, making it particularly effective for longer code contexts and complex programming tasks.
Q: What are the recommended use cases?
This model is ideal for code generation, code completion, programming assistance, and handling programming-related queries across multiple languages. It's particularly useful in educational contexts and for developers seeking AI-powered coding assistance.
