bert-base-polish-uncased-v1
| Property | Value |
|---|---|
| Architecture | BERT Base (12-layer, 768-hidden, 12-heads) |
| Parameters | 110M |
| Training Data | 1.86B words from Polish corpora |
| Author | dkleczek |
What is bert-base-polish-uncased-v1?
This is a Polish language model based on BERT architecture, specifically trained on a large corpus of Polish text. It represents a significant milestone in Polish NLP, providing a powerful tool for various language understanding tasks. The model was trained on a diverse dataset including Polish Wikipedia, Parliamentary Corpus, ParaCrawl, and Open Subtitles, totaling over 1.86 billion words.
Implementation Details
The model follows the bert-base architecture with 12 transformer layers, 768 hidden dimensions, and 12 attention heads. Training was conducted over 1 million steps using Google Cloud TPU v3-8, with varying sequence lengths and batch sizes. The training process included 100,000 steps with 128 sequence length and batch size 512, followed by 800,000 steps with adjusted parameters, and finally 100,000 steps with 512 sequence length.
- Comprehensive training on Polish-specific text corpora
- Optimized for uncased input processing
- Supports masked language modeling capabilities
- Achieves strong performance on KLEJ benchmark tasks
Core Capabilities
- Text Classification
- Masked Language Modeling
- Named Entity Recognition (93.6% accuracy on NKJP-NER)
- Sentiment Analysis
- General Polish language understanding tasks
Frequently Asked Questions
Q: What makes this model unique?
This model is specifically optimized for Polish language processing, trained on a massive corpus of Polish text, making it particularly effective for Polish-specific NLP tasks. It achieves strong performance on the KLEJ benchmark, demonstrating its effectiveness across various language understanding tasks.
Q: What are the recommended use cases?
The model is well-suited for tasks including text classification, named entity recognition, and sentiment analysis in Polish text. It performs particularly well on tasks requiring deep language understanding and can be fine-tuned for specific domain applications.
