weblab-10b
| Property | Value |
|---|---|
| Author | Takeshi Kojima (matsuo-lab) |
| Model Size | 10 billion parameters |
| License | cc-by-nc-4.0 |
| Architecture | 36-layer, 4864-hidden-size transformer |
What is weblab-10b?
weblab-10b is a powerful Japanese-centric multilingual language model based on the GPT-NeoX architecture. Trained on an impressive 600B tokens from Japanese C4 and The Pile datasets, it represents a significant advancement in Japanese language processing capabilities. The model demonstrates strong performance across various Japanese language tasks, particularly in the JGLUE benchmark suite.
Implementation Details
Built on EleutherAI's GPT-NeoX framework, this model features a sophisticated 36-layer architecture with a 4864 hidden size. It's designed for efficient processing of both Japanese and English text, making it particularly valuable for multilingual applications. The model can be easily implemented using PyTorch and the Transformers library.
- Trained on 600B tokens from diverse sources
- Implements GPT-NeoX architecture with 36 layers
- Supports both Japanese and English text generation
- Available in float16 precision for efficient inference
Core Capabilities
- Strong performance on JGLUE benchmark (50.74% average across 8 tasks)
- Exceptional results in MARC-ja task (82.07% accuracy)
- Robust performance in JSQuAD (62.94% accuracy)
- Effective text generation in both Japanese and English
Frequently Asked Questions
Q: What makes this model unique?
weblab-10b stands out for its specialized focus on Japanese language processing while maintaining multilingual capabilities. Its large-scale training on 600B tokens and strong benchmark performance make it particularly suitable for Japanese language tasks.
Q: What are the recommended use cases?
The model is ideal for Japanese text generation, question answering, and natural language understanding tasks. It's particularly effective for applications requiring strong performance on JGLUE benchmark tasks like MARC-ja and JSQuAD.
