BgGPT-Gemma-2-2.6B-IT-v1.0
| Property | Value |
|---|---|
| Parameter Count | 2.61B |
| Model Type | Causal decoder-only transformer |
| Base Model | google/gemma-2-2b-it |
| License | Gemma Terms of Use |
| Languages | Bulgarian, English |
What is BgGPT-Gemma-2-2.6B-IT-v1.0?
BgGPT-Gemma-2-2.6B-IT-v1.0 is a state-of-the-art Bulgarian language model developed by INSAIT Institute. Built on Google's Gemma 2 2B architecture, it was pre-trained on approximately 100 billion tokens, with 85 billion in Bulgarian, using an innovative Branch-and-Merge strategy. This model represents a significant advancement in Bulgarian language AI while maintaining strong English language capabilities.
Implementation Details
The model utilizes a BF16 tensor type and implements a sophisticated pre-training approach that combines Bulgarian web crawl data, Wikipedia content, and specialized datasets. The model has been instruction-fine-tuned using real-world Bulgarian conversations.
- Continuous pre-training using Branch-and-Merge strategy
- Instruction fine-tuning on Bulgarian conversation dataset
- Optimized generation parameters with temperature=0.1 and top_k=25
- Supports both Bulgarian and English language processing
Core Capabilities
- Strong performance on standard benchmarks including Winogrande, Hellaswag, and ARC
- Excels in Bulgarian-specific tasks and cultural context
- Maintains comparable English language performance to base Gemma model
- Supports conversational AI applications
- Handles complex reasoning and knowledge-based tasks
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its exceptional Bulgarian language capabilities while maintaining English performance, achieved through an innovative Branch-and-Merge training strategy and extensive pre-training on Bulgarian content.
Q: What are the recommended use cases?
The model is ideal for Bulgarian-English bilingual applications, including text generation, conversation, educational assistance, and general language understanding tasks. It's particularly effective for applications requiring strong cultural and linguistic understanding of Bulgarian context.
