Llama-3.2-Taiwan-3B-Instruct
| Property | Value |
|---|---|
| Model Type | LlamaForCausalLM |
| Languages | Traditional Chinese (zh-tw), English |
| License | llama3.2 |
| Base Model | lianghsun/Llama-3.2-Taiwan-3B |
What is Llama-3.2-Taiwan-3B-Instruct?
Llama-3.2-Taiwan-3B-Instruct is a specialized language model fine-tuned specifically for Traditional Chinese and Taiwanese context. Built upon the Llama architecture, this 3B parameter model has undergone instruction fine-tuning and direct preference optimization (DPO) using extensive Traditional Chinese dialogue datasets and multilingual conversation collections.
Implementation Details
The model was trained using a multi-GPU setup with 4 NVIDIA H100 NVL cards, implementing a sophisticated training procedure that includes both SFT (Supervised Fine-Tuning) and DPO stages. The training process utilized Adam optimizer with carefully tuned hyperparameters and a cosine learning rate scheduler.
- Training Duration: 5+ days
- Batch Size: 21,000
- Learning Rate: 5e-05 to 5e-07
- Training FLOPs: 26.66T
Core Capabilities
- Advanced Traditional Chinese language understanding and generation
- Specialized knowledge of Taiwanese legal system and culture
- Multi-turn dialogue capabilities
- Domain expertise in various professional fields including law, medicine, and technology
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its specialized focus on Traditional Chinese and Taiwanese context, making it particularly effective for applications requiring deep understanding of Taiwan-specific cultural, legal, and professional contexts. It has been trained on diverse datasets including legal, medical, and technical content specific to Taiwan.
Q: What are the recommended use cases?
The model is well-suited for applications requiring Traditional Chinese language processing, particularly in professional contexts such as legal document analysis, technical documentation, and general Taiwanese cultural context. It's designed for direct deployment in inference endpoints and can be further fine-tuned for specific domain applications.
