CS336: Language Modeling from Scratch by Stanford

Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment.

Table of contents:
1. Foundations
1.1 Overview
1.2 Tokenization
1.3 PyTorch & einops
1.4 Resource Accounting
- FLOPs
- Memory
- Arithmetic Intensity
2. Transformer Architecture
2.1 Model Architectures
2.2 Hyperparameters
2.3 Attention Alternatives
2.4 Mixture of Experts (MoE)
3. Systems for Training
3.1 GPUs and TPUs
3.2 GPU Kernels
3.3 Triton
3.4 Parallelism
- Part I
- Part II
4. Scaling
4.1 Scaling Laws
- Theory
- Applications
5. Inference
6. Evaluation
7. Data
7.1 Data Sources
7.2 Datasets
7.3 Filtering
7.4 Deduplication
7.5 Data Mixing
7.6 Synthetic Data
8. Post-Training
8.1 Supervised Fine-Tuning (SFT)
8.2 RLHF / Mid-Training
8.3 RLVR
9. Alignment
9.1 Multimodal Alignment
10. Guest Lectures
10.1 Daniel Selsam
10.2 Dan Fu

Links:
• Site
• YouTube

Navigational hashtags: #armknowledgesharing #armcourses
General hashtags: #llm #largelanguagemodels #stanford

@data_science_weekly