financial_roberta
| Property | Value |
|---|---|
| Parameter Count | 86.6M |
| Model Type | RoBERTa MLM |
| Architecture | Transformer-based with 6 hidden layers, 12 attention heads |
| Paper | RoBERTa Paper |
What is financial_roberta?
financial_roberta is a specialized masked language model based on the RoBERTa architecture, specifically trained on the Financial Phrasebank corpus. This model is designed to understand and process financial text, making it particularly useful for financial NLP tasks.
Implementation Details
The model utilizes a RoBERTa architecture with 86.6M parameters, featuring 6 hidden layers and 12 attention heads. It was trained with a vocabulary size of 56,000 tokens and maximum position embeddings of 514. The training process involved 10 epochs with a GPU batch size of 64 units.
- Vocabulary size: 56,000 tokens
- Maximum position embeddings: 514
- Hidden layers: 6
- Attention heads: 12
- Type vocabulary size: 1
Core Capabilities
- Masked language modeling for financial text
- Financial sentiment analysis
- Context-aware text completion in financial documents
- Understanding of financial terminology and context
Frequently Asked Questions
Q: What makes this model unique?
This model's uniqueness lies in its specialized training on financial text data from the Financial Phrasebank corpus, making it particularly adept at understanding financial context and terminology. The model's architecture is optimized for financial language processing while maintaining a relatively compact size of 86.6M parameters.
Q: What are the recommended use cases?
The model is ideal for financial text analysis tasks such as: Predicting missing words in financial statements, Understanding financial sentiment, Analyzing company reports and filings, Processing financial news and updates.
