bert-tiny-historic-multilingual-cased
| Property | Value |
|---|---|
| Parameter Count | 4.62M |
| License | MIT |
| Paper | Well-Read Students Learn Better |
| Languages | German, French, English, Finnish, Swedish |
What is bert-tiny-historic-multilingual-cased?
This is a compact multilingual BERT model specifically designed for processing historical texts. It's trained on a diverse corpus of historical documents from five European languages, with special attention to OCR-processed texts from various sources including Europeana and the British Library.
Implementation Details
The model features a tiny architecture with 2 layers and 128 hidden dimensions, resulting in 4.58M parameters. It was trained on a combined corpus of 130GB of text data, with each language contributing approximately 24-28GB. The training process utilized Google's TPU Research Cloud, achieving efficient training speeds of 4.3 seconds per 1,000 steps.
- Vocabulary size: 32k tokens with extremely low unknown token rates (0.0001-0.0007)
- Supports maximum sequence length of 512 tokens
- Trained with carefully filtered OCR content (confidence threshold ≥ 0.6)
Core Capabilities
- Multilingual processing of historical texts from the 19th century
- Efficient masked language modeling for 5 European languages
- Optimized for historical document understanding
- Balanced performance across all supported languages
Frequently Asked Questions
Q: What makes this model unique?
This model is specifically optimized for historical text processing, combining efficiency with multilingual capabilities. Its tiny architecture makes it particularly suitable for resource-constrained environments while maintaining effectiveness across five European languages.
Q: What are the recommended use cases?
The model is ideal for processing historical documents, particularly those from the 19th century. It's especially useful for digital humanities projects, historical research, and applications involving OCR-processed historical texts in German, French, English, Finnish, or Swedish.
