Knowledge Distillation (KD)

Techniques for compressing models by transferring knowledge from teacher to student models.

Key Concepts

• Student-Teacher Framework
• Soft Targets
• Temperature Scaling

To Explore

• Dark Knowledge
• Response-based vs Feature-based Distillation
• Cross-modal Distillation