Compact-biobert
| Property | Value |
|---|---|
| Parameters | 65 million |
| License | MIT |
| Author | nlpie |
| Downloads | 33,559 |
| Tags | Fill-Mask, Transformers, PyTorch, BERT |
What is compact-biobert?
Compact-biobert is a highly efficient, distilled version of the original BioBERT model, specifically designed for biomedical text processing. It maintains high performance while reducing computational requirements through sophisticated distillation techniques trained on the PubMed dataset for 100,000 steps.
Implementation Details
The model implements a unique architecture combining approaches from DistilBioBERT and TinyBioBERT. It features 6 transformer layers with a hidden dimension of 768 and a vocabulary size of 28,996. The model uses an innovative layer-to-layer distillation process incorporating MLM, layer, and output distillation components.
- Hidden dimension: 768
- Transformer layers: 6
- Feed-forward expansion rate: 4x
- Vocabulary size: 28,996
Core Capabilities
- Biomedical text processing and analysis
- Masked language modeling for biomedical content
- Efficient processing with reduced computational requirements
- Maintains performance while reducing model size
Frequently Asked Questions
Q: What makes this model unique?
This model uniquely combines distillation approaches from both DistilBioBERT and TinyBioBERT, using a specialized initialization technique where weights are taken from every other layer of the teacher model.
Q: What are the recommended use cases?
The model is particularly suited for biomedical text processing tasks where computational efficiency is important, such as large-scale biomedical document analysis, medical text mining, and clinical NLP applications.
