Architecture Variants

Exploration of different transformer architectures and design choices.

Key Concepts

• Decoder-only vs Encoder-Decoder
• ALiBi and positional encoding
• Normalization strategies

To Explore

• MQA / GQA (grouped query attention)
• Hybrid architectures
• State space models