Learning Rate Schedules
Techniques for optimizing learning rate schedules during training.
Key Strategies
Section titled “Key Strategies”- Cosine Annealing
- Warmup Phases
- Cyclical Learning Rates
To Explore
Section titled “To Explore”- Schedule selection by task
- Interaction with batch size
- Distributed training adaptations