StripedHyena-Nous-7B
| Property | Value |
|---|---|
| Parameter Count | 7.65B |
| License | Apache 2.0 |
| Context Length | 32k tokens |
| Architecture Type | Hybrid (Attention + Convolution) |
| Precision | FP16 |
What is StripedHyena-Nous-7B?
StripedHyena-Nous-7B (SH-N 7B) is an innovative chat model developed through collaboration between Together Research and Nous Research. It represents a significant departure from traditional Transformer architectures, combining multi-head grouped-query attention with gated convolutions in Hyena blocks. This model is particularly notable for being the first alternative architecture to achieve competitive performance with leading open-source Transformers.
Implementation Details
The model employs a sophisticated hybrid architecture that leverages signal processing principles alongside machine learning techniques. It features constant memory decoding in Hyena blocks, implemented through state-space models or truncated filters, enabling efficient processing of longer sequences.
- Utilizes constant memory decoding via state-space model representation
- Implements multi-head, grouped-query attention mechanisms
- Features gated convolutions arranged in Hyena blocks
- Supports sequence lengths up to 32k tokens
- Operates in FP16 precision with specific float32 requirements for poles and residues
Core Capabilities
- Enhanced throughput and lower latency compared to traditional Transformers
- Superior training and inference-optimal scaling laws vs. Llama-2
- Efficient long-context processing
- Optimized for chat interactions with a specific prompt format
- Improved memory efficiency during inference
Frequently Asked Questions
Q: What makes this model unique?
The model's hybrid architecture combining signal processing principles with traditional ML approaches sets it apart, offering competitive performance while requiring less computational resources than conventional Transformers.
Q: What are the recommended use cases?
StripedHyena-Nous-7B is optimized for chat applications, particularly those requiring processing of longer context windows up to 32k tokens. It's ideal for applications needing efficient, high-throughput text generation with maintained quality.
