Softmax vs Hardmax by hand ✍️ ~ interactive calculator 👉 https://byhand.ai/vhUJDH

Softmax turns a set of raw scores (z) into a probability distribution (Y) over choices (a, b, c, d, e). Instead of just saying which option is best, it tells us how likely each option is to be chosen. In this example, most of the probability mass is concentrated on c, while the other options are still possible but clearly less likely. That's the point of softmax: it converts relative scores into meaningful, comparable probabilities that sum to 100%.

Think of a raffle. Hardmax is when the person who bought the most tickets always wins the prize — the top score takes it, every time. Softmax is when everyone's chance is proportional to the tickets they hold: even if I bought just one ticket, I may still get lucky. Who knows. That's the psychology of softmax.

This is how a language model chooses its next word. Each time a word appears in the training data, it earns a ticket. Hardmax would always speak the word with the most tickets — the same safe choice, over and over. Softmax gives every word a chance proportional to its tickets, so less common words can still be spoken. The word with the most tickets still has the highest chance of winning — just not 100%. That's what lets the model surprise us with its creativity (and also its hallucinations) instead of repeating itself.

https://t.me/CodeProgrammer 😱