StableLM-Tuned-Alpha-3B
| Property | Value |
|---|---|
| Parameter Count | 3 Billion |
| Architecture | NeoX Transformer |
| Hidden Size | 4096 |
| Layers | 16 |
| Attention Heads | 32 |
| License | CC BY-NC-SA-4.0 |
What is stablelm-tuned-alpha-3b?
StableLM-Tuned-Alpha-3B is a sophisticated decoder-only language model developed by Stability AI, specifically designed for chat and instruction-following tasks. Built upon the StableLM-Base-Alpha architecture, this model has been fine-tuned on a diverse collection of high-quality datasets to enhance its conversational abilities and instruction-following capabilities.
Implementation Details
The model implements a NeoX transformer architecture with a 4096 sequence length capability. It's trained using mixed-precision (FP16) and optimized with AdamW, using a batch size of 256, learning rate of 2e-5, and 50 warm-up steps. The training process incorporated weight decay of 0.01 and beta values of (0.9, 0.99).
- Utilizes a specialized stopping criteria system for generating responses
- Implements a structured prompt format: <|SYSTEM|>...<|USER|>...<|ASSISTANT|>...
- Trained on multiple high-quality datasets including Alpaca, GPT4All, Anthropic HH, DataBricks Dolly, and ShareGPT Vicuna
Core Capabilities
- Enhanced chat and dialogue generation
- Instruction following and task completion
- Poetry and creative writing
- Question answering and information provision
- Safe and ethical response generation
Frequently Asked Questions
Q: What makes this model unique?
The model's distinctive feature is its comprehensive fine-tuning on five diverse datasets, combined with its focus on safe and helpful interactions. It's specifically designed to refuse harmful requests while maintaining creative capabilities.
Q: What are the recommended use cases?
The model is ideal for chat applications, creative writing tasks, and general instruction following. It's particularly suited for non-commercial applications requiring a balance of helpfulness and safety in AI interactions.
