dolphin-llama-13b
| Property | Value |
|---|---|
| Base Model | LLaMA 13B |
| License | Non-commercial use only |
| Training Data | ~3.5M instructions (GPT-3.5 & GPT-4 completions) |
| Average Benchmark Score | 53.56% |
What is dolphin-llama-13b?
Dolphin-llama-13b is an uncensored language model based on LLaMA architecture, implementing Microsoft's Orca approach to progressive learning. The model was trained on a carefully curated dataset combining GPT-4 and GPT-3.5 completions, with specific focus on removing alignment bias and censorship.
Implementation Details
The model underwent a two-phase training process on 8x H100 GPUs for approximately 600 hours. The first phase used 2.6M GPT-3.5 completions (3 epochs, lr=2e-5), followed by 842K GPT-4 completions (2.5 epochs, lr=1e-5). It implements a Vicuna-style prompt format with added system instructions.
- Extensive dataset cleaning and deduplication
- Filtered alignment and bias instances
- Custom prompt format with SYSTEM field
- Comprehensive benchmark evaluation
Core Capabilities
- Strong performance on HellaSwag (77.11% accuracy)
- Solid reasoning capabilities (Winogrande: 69.93%)
- Balanced truthfulness (TruthfulQA: 52.23%)
- Comprehensive knowledge testing (MMLU: 52.16%)
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its uncensored nature and implementation of Microsoft's Orca methodology, offering high compliance while allowing users to implement their own alignment layer.
Q: What are the recommended use cases?
The model is suitable for research and non-commercial applications requiring unrestricted outputs, though users should implement appropriate safety measures before deployment.
