🧠 LayerNorm vs BatchNorm: Same Goal, Different Behavior

Both techniques normalize activations, but they operate differently.

Batch Normalization

📦 Normalizes across the batch
⚡️ Depends on batch statistics
🖼 Works very well in CNNs
⚠️ Sensitive to small batch sizes

Layer Normalization

🔬 Normalizes across features per sample
📏 Independent of batch size
🤖 Preferred in transformers and NLP
✅ Stable for sequence models

Why transformers use LayerNorm❔

Sequence models often run with variable or small batches.
LayerNorm avoids reliance on batch statistics and stays stable.

✅ Rule of thumb

🖼 CNNs → BatchNorm
🤖 Transformers → LayerNorm

📌 They look similar mathematically but normalize along different axes.