👍 Learn all about LLM in 30 minutes. This is the best explanation of what's happening inside ChatGPT

People believe that ChatGPT deeply understands everything it writes and is even better at creating stories. In reality, the transformer model predicts the most likely word to come next in the text.

Stanford graduate Grant Sanderson has released a 30-minute video explaining how GPT works. Below is a brief FAQ about the word selection mechanism. Spoiler: We don't promise it will be easy, but it will definitely be simpler than textbooks.

🔠 The text generation process can be described in 3 words: Prediction → Addition → Repetition. LLM models predict the next word in the text, add it to the text, and then predict the word again based on the resulting text.

🔠 In the first stage, the model divides the text into tokens: words, parts of words, and combinations of characters.

🔠 The model determines the meaning of each word based on context and the word's position in the phrase. This is done using an attention mechanism. Tokens can "communicate" with each other at the level of numbers to refine the context. As a result of this "communication," the data for each token is updated.

🔠 The next stage involves the model asking a set of questions about each token: is it a noun, what language is it written in, etc? This stage is handled by a multilayer perceptron.

🔠 After numerous repetitions of these two operations, a list of words appears that are likely to follow in the text. Each word is assigned a probability, and the model selects the word with the highest probability. Then, the transformer starts looking for the next word, and so on.

⏰ Lecture timestamps:

⏺️ Process of finding a word.
⏺️ Main idea of Deep ML.
⏺️ Where the sought-after word is located.
⏺️ How words turn into vectors.
⏺️ Why ChatGPT forgets conversation context.

▶️ Watch the full review of transformers here.

Also related:

🤪 Why Does Generative AI Make Mistakes?

🔍 Michael Wooldridge: What's the Future for Generative AI?

@hiaimediaen