Stanford packed 1.5 hours with everything you need to know about LLMs

Here are 5 lessons that stood out from the lecture:

1/ Architecture ≠ Everything
→ Transformers aren’t the bottleneck anymore.
→ In practice, data quality, evaluation design, and system efficiency drive real gains.

2/ Tokenizers Are Underrated
→ A single tokenization choice can break performance on math, code, or logic.
→ Most models can't generalize numerically because 327 might be one token, while 328 is split.

3/ Scaling Laws Guide Everything
→ More data + bigger models = better loss. But it's predictable.
→ You can estimate how much performance you’ll gain before you even train.

4/ Post-training = The Real Upgrade
→ SFT teaches the model how to behave like an assistant.
→ RLHF and DPO tune what it says and how it says it.

5/ Training is 90% Logistics
→ The web is dirty. Deduplication, PII filtering, and domain weighting are massive jobs.
→ Good data isn’t scraped, it’s curated, reweighted, and post-processed for weeks.