Skip to content
SJ
The Scaling Journey
Search
Ctrl
K
Cancel
GitHub
LinkedIn
Substack
Select theme
Dark
Light
Auto
Writing
Overview
Essays
Getting Started with The Scaling Journey
Notes
Notes
LLMs
Overview
Troiani
Overview
Architecture
Architecture Variants
Attention Mechanisms
Scaling
Compute Efficiency
Power Laws in LLMs
Pre-training
Architecture
Batch Normalization Tricks
Data
Learning Rate Schedules
Optimization
Mid-training
Fine-tuning
Instruction Tuning
RLHF
Tools & Agents
Compression
Knowledge Distillation (KD)
Pruning
Quantization-Aware Training (QAT)
Quantization
Restoration Process
Retrieval & RAG
Embeddings & Indexing
RAG Systems
World Models
Overview
Foundations
Prediction
Dynamics Models
Visual Prediction
Workloads
Evaluation
Research notes
AI Infra
Overview
Performance
Training systems
Inference systems
Cluster & platform
Data plane
Kernels & compilers
Deployment
World Model Platform
Harness
Overview
Advertisement
Overview
Notes
The Price of Attention
Me
About
CV
Work
Projects
Old site
GitHub
LinkedIn
Substack
Select theme
Dark
Light
Auto
Knowledge Distillation (KD)
WIP.