Codestral-22B-v0.1-GGUF
| Property | Value |
|---|---|
| Parameter Count | 22.2B parameters |
| License | MNPL |
| Author | bartowski |
| Base Model | mistralai/Codestral-22B-v0.1 |
| Quantization Types | Multiple (Q8_0 to IQ2_XS) |
What is Codestral-22B-v0.1-GGUF?
Codestral-22B-v0.1-GGUF is a sophisticated code-generation model that offers various GGUF quantizations of the original Mistral AI's Codestral-22B model. It's specifically optimized for programming tasks and comes in multiple compression levels to accommodate different hardware configurations and performance requirements.
Implementation Details
The model uses llama.cpp for quantization and offers 17 different versions ranging from 23.64GB (Q8_0) to 6.64GB (IQ2_XS). Each version represents a different trade-off between model size and performance. The implementation supports various quantization techniques including both traditional K-quants and newer I-quants, optimized for different hardware configurations.
- Supports specific prompt format with system prompts
- Compatible with cuBLAS (Nvidia) and rocBLAS (AMD)
- Offers both high-quality and resource-efficient versions
- Implements SOTA quantization techniques
Core Capabilities
- Code generation and completion
- Multiple quantization options for different hardware setups
- Flexible deployment options from high-end to resource-constrained environments
- Support for various inference engines
Frequently Asked Questions
Q: What makes this model unique?
The model stands out for its comprehensive range of quantization options, allowing users to choose the perfect balance between model size and performance. It's particularly notable for incorporating both traditional K-quants and newer I-quants, making it versatile across different hardware configurations.
Q: What are the recommended use cases?
This model is ideal for code-related tasks. For optimal performance, users with high-end GPUs should consider the Q6_K or Q5_K_M versions, while those with limited resources might opt for the IQ4_XS or lower versions which still maintain reasonable performance.
