ESM-1b Protein Language Model
| Property | Value |
|---|---|
| Model Size | 650M parameters |
| License | MIT |
| Training Data | Uniref50 (2018_03) |
| Paper | Link |
What is esm1b_t33_650M_UR50S?
ESM-1b is a sophisticated transformer-based protein language model developed by Facebook Research. Built on the RoBERTa architecture, it's designed for understanding and predicting protein sequences through unsupervised learning on approximately 30 million protein sequences.
Implementation Details
The model utilizes pre-activation layer normalization and is trained using masked language modeling on sequences up to 1023 tokens long. Training was conducted on 128 NVIDIA V100 GPUs for 500K updates, using Adam optimizer with specific learning rate scheduling.
- Masked language modeling with 15% masking rate
- Sequence length of 1024 tokens (including CLS token)
- Vocabulary size of 21 amino acids
- Trained on Uniref50 2018_03 dataset
Core Capabilities
- Zero-shot transfer to structure and function prediction
- Contact prediction through attention heads
- Mutation effect prediction
- Feature extraction for downstream tasks
Frequently Asked Questions
Q: What makes this model unique?
ESM-1b stands out for its ability to learn protein structure and function relationships without explicit supervision, achieving state-of-the-art results in various protein analysis tasks.
Q: What are the recommended use cases?
The model is ideal for protein sequence analysis, structure prediction, and function prediction. It can be used for feature extraction, fine-tuning on specific tasks, or direct inference about protein properties.
