DeepScaleR-1.5B-Preview GGUF
| Property | Value |
|---|---|
| Original Model | agentica-org/DeepScaleR-1.5B-Preview |
| Size Range | 0.77GB - 7.11GB |
| Quantization Types | Multiple (F32 to IQ3_XXS) |
| Author | bartowski |
What is agentica-org_DeepScaleR-1.5B-Preview-GGUF?
This is a comprehensive collection of GGUF quantized versions of the DeepScaleR-1.5B model, optimized for various hardware configurations and use cases. The collection offers different compression levels while maintaining different quality-size tradeoffs, ranging from full F32 weights (7.11GB) to highly compressed versions (0.77GB).
Implementation Details
The model uses a specific prompt format: <｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜><｜end▁of▁sentence｜><｜Assistant｜>. It implements various quantization techniques, including new methods like IQ4_NL and IQ3_M for improved performance on specific hardware.
- Supports multiple quantization types including Q8_0, Q6_K, Q5_K, Q4_K, Q3_K, and IQ series
- Features online repacking for ARM and AVX CPU inference optimization
- Implements special handling for embed/output weights in certain variants
Core Capabilities
- Flexible deployment options for different hardware configurations
- Optimized performance for both CPU and GPU implementations
- Support for multiple inference engines including LM Studio and llama.cpp
- Special optimizations for ARM and AVX architectures
Frequently Asked Questions
Q: What makes this model unique?
The model offers an extensive range of quantization options, allowing users to choose the perfect balance between model size, quality, and performance for their specific hardware setup. It particularly stands out for its implementation of newer quantization methods like the IQ series.
Q: What are the recommended use cases?
For maximum quality, use Q6_K_L or Q8_0 variants. For balanced performance, Q4_K_M is recommended. For low RAM systems, consider Q3_K_XL or IQ3_M variants. When using ARM or AVX systems, Q4_0 or IQ4_NL variants are particularly effective.
