BARThez
| Property | Value |
|---|---|
| Base Model Parameters | 165M |
| mBARThez Parameters | 458M |
| Training Data | 66GB French text |
| Paper | arXiv:2010.12321 |
| GitHub | Repository |
What is BARThez?
BARThez is a French sequence-to-sequence model based on the BART architecture, specifically designed for generative tasks in the French language. It comes in two variants: BARThez (trained from scratch) and mBARThez (adapted from multilingual BART). The model is pretrained on a massive 66GB corpus of French text using a reconstruction objective where it learns to recover corrupted input sentences.
Implementation Details
The model implements two distinct architectures: a BASE version with 12 layers (BARThez) containing 165M parameters, and a LARGE version with 24 layers (mBARThez) containing 458M parameters. The pretraining process involves reconstructing corrupted input text, making it particularly effective for generative tasks.
- BARThez BASE: 12-layer architecture with 165M parameters
- mBARThez LARGE: 24-layer architecture with 458M parameters
- Pretrained on 66GB of French text data
- Implements both encoder and decoder pretraining
Core Capabilities
- Specialized in French language generation tasks
- Particularly effective for abstractive summarization
- Superior performance in generative tasks compared to BERT-based models
- Fully pretrained encoder and decoder components
- Adaptable for various sequence-to-sequence tasks
Frequently Asked Questions
Q: What makes this model unique?
Unlike other French language models like CamemBERT and FlauBERT that are BERT-based, BARThez features both pretrained encoder and decoder components, making it specifically suited for generative tasks. The availability of two variants (BARThez and mBARThez) offers flexibility for different use cases and computational requirements.
Q: What are the recommended use cases?
BARThez is particularly well-suited for generative tasks in French, especially abstractive summarization. It can be effectively used for any sequence-to-sequence task requiring French language generation, including text generation, translation, and text reformulation.
