GPT2-Finetuned-Greek
| Property | Value |
|---|---|
| Parameter Count | 117M |
| License | Apache-2.0 |
| Training Data Size | 23.4GB |
| Perplexity | 39.12 |
| Base Architecture | GPT-2 (12-layer, 768-hidden, 12-heads) |
What is gpt2-finetuned-greek?
GPT2-finetuned-greek is an advanced language model specifically adapted for the Greek language, developed through a collaboration between the Hellenic Army Academy (SSE) and the Technical University of Crete (TUC). It represents a sustainable and efficient approach to creating language models for low-resource languages by fine-tuning the original English GPT-2 model rather than training from scratch.
Implementation Details
The model utilizes the GPT-2 architecture with 12 transformer layers, 768 hidden dimensions, and 12 attention heads, totaling 117M parameters. The training process employed gradual layer unfreezing and was implemented using Huggingface transformers and fastai frameworks. The model was trained on a diverse 23.4GB Greek corpus compiled from multiple sources including CC100, Wikimatrix, Tatoeba, Books, SETIMES, and GlobalVoices.
- Pre-processing includes tokenization and BPE segmentation
- Achieves a training loss of 3.67 and validation loss of 3.83
- Implements repetition penalty and temperature controls for better text generation
Core Capabilities
- Autoregressive text generation in Greek language
- Supports customizable generation parameters (temperature, top-k, top-p)
- Optimized for long-sequence processing
- Efficient inference with controllable output length
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its efficient approach to Greek language modeling, using transfer learning instead of training from scratch. It's particularly notable for achieving strong performance (39.12 perplexity) while being resource-efficient in its training approach.
Q: What are the recommended use cases?
The model is well-suited for Greek text generation tasks, creative writing assistance, content generation, and as a foundation for further Greek language processing applications. It's particularly effective for generating coherent, contextually-appropriate Greek text with customizable generation parameters.
