Alpaca Native
| Property | Value |
|---|---|
| Author | chavinlo |
| Framework | PyTorch |
| Training Infrastructure | 4xA100 GPUs |
| Training Duration | 6 Hours |
| Benchmark Score | 41.96 (OpenLLM) |
What is alpaca-native?
Alpaca-native is a natively fine-tuned implementation of Stanford's Alpaca model, distinguished by its direct fine-tuning approach without using LoRA adaptation techniques. The model leverages the LLaMA architecture and was trained using FSDP (Fully Sharded Data Parallel) mode for efficient distributed training.
Implementation Details
The model was trained using a specific configuration optimized for distributed training across 4 A100 GPUs. It employs BF16 precision training with cosine learning rate scheduling and gradient accumulation steps of 8. The training process utilized a learning rate of 2e-5 and included a 3% warmup ratio.
- Native fine-tuning without LoRA
- FSDP implementation for distributed training
- Optimized hyperparameters for stability
- Wandb integration for monitoring
Core Capabilities
- ARC Performance: 52.3 (25-shot)
- HellaSwag: 77.09 (10-shot)
- MMLU: 41.6 (5-shot)
- TruthfulQA: 37.58 (0-shot)
- Winogrande: 69.46 (5-shot)
Frequently Asked Questions
Q: What makes this model unique?
The model's native fine-tuning approach, eschewing LoRA, sets it apart from other Alpaca implementations. This approach potentially offers more robust performance at the cost of increased training resources.
Q: What are the recommended use cases?
Given its benchmark performance, the model is well-suited for general text generation tasks, particularly those requiring strong performance on common sense reasoning (HellaSwag) and reading comprehension (ARC). However, its relatively lower performance on GSM8K suggests caution for mathematical reasoning tasks.
