| Property | Value |
|---|---|
| Parameter Count | 2.07M |
| Model Type | Text Generation |
| Architecture | Phi-3 (Minimal) |
| Tensor Type | F32 |
| Downloads | 325,708 |
What is tiny-random-Phi3ForCausalLM?
tiny-random-Phi3ForCausalLM is a minimalistic implementation of the Phi-3 architecture, developed by Xenova. This compact model features just 2.07M parameters, making it ideal for experimentation and testing purposes. The model is built with a simplified architecture consisting of 2 hidden layers, 4 attention heads, and maintains a sliding window of 2047 tokens.
Implementation Details
The model implements a streamlined version of the Phi-3 architecture with the following specifications: hidden size of 32, intermediate size of 64, and 4 key-value heads. It utilizes the microsoft/Phi-3-mini-4k-instruct tokenizer and is optimized for F32 tensor operations.
- Minimal architecture with 2 hidden layers
- 4 attention heads for efficient processing
- Sliding window of 2047 tokens
- 32 hidden size dimensions
- 64 intermediate size dimensions
Core Capabilities
- Basic text generation tasks
- Experimental model testing
- Lightweight deployment scenarios
- Educational purposes and architecture study
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its extremely compact size while maintaining the core Phi-3 architecture components. It's particularly useful for testing and educational purposes where a full-scale model would be unnecessary.
Q: What are the recommended use cases?
The model is best suited for experimental implementations, proof-of-concept testing, and educational scenarios where understanding the basic mechanics of transformer-based models is the primary goal.