Have you ever wondered why first token always appears with a delay, while rest of stream proceeds almost instantly?
It's not network latency or model warmup, it's a structural property of how LLMs actually execute.
Inference consists of two phases that use the same model and execution path, but the workload in each phase is fundamentally different, and the bottlenecks are opposite.
𝗣𝗿𝗲𝗳𝗶𝗹𝗹 — this is the request processing phase. The model processes all input tokens in a single parallel pass, computing Q, K, and V for all tokens at once.
The attention mechanism is implemented as a large matrix operation, for which GPUs are optimized, so the computational units are heavily loaded and the chip operates at the limit of its arithmetic throughput.
Pre-processing is memory-bound, and the metric that reflects this is the time to the first token.
𝗗𝗲𝗰𝗼𝗱𝗲 starts after the first token appears. To generate the next one, the model calculates Q, K, and V only for the new token, because everything previous is already cached.
Then comes the "one token - one pass" cycle: the new query is multiplied by the already stored keys instead of the full matrix, and the computational volume becomes small.
However, the GPU still has to read all weights and the entire cache from memory to perform even this small operation, so the memory bandwidth becomes the bottleneck, and the computational units are idle.
Decoding is memory-bound, and the metric here is the delay between tokens.
This separation explains a number of effects that seem non-obvious from the outside.
The GPU load is high during pre-processing and drops sharply during decoding, because in the second phase, the memory becomes the limiting factor rather than the computations.
Adding computational power often doesn't help with slow generation, because for memory-bound workloads, the solution is faster memory or a smaller cache, not more FLOPs.
A long context slows down generation disproportionately, because the key and value caches grow with each token, and each step of decoding must read them all.
This cache is a key optimization, without which decoding would be impossible, because the attention would have to be recalculated for the entire growing sequence at each step.
With the cache, it's built once during pre-processing and then expanded by one element for each new token, reusing already computed values.
However, the cache is stored in GPU memory and grows linearly with the sequence length. For a 13B model, this is about 1 MB per token, so a 4K context occupies about 4 GB of video memory just for the cache.
Therefore, a long context feels slow not because of the "lack of model power," but because of the memory pressure.
Currently, the industry is optimizing this limitation through quantized caches, sliding windows, grouped attention, and PagedAttention, while the DeepSeek V4 series goes further and redesigns the attention mechanism itself to make the cache smaller from the start.
When attention starts being redesigned for memory constraints, it means that the limitation has shifted towards memory.
Practical takeaway: if the model seems slow, it's important to distinguish — is it slow starting or slow streaming? A slow start corresponds to pre-processing and computational bottlenecks, while slow streaming corresponds to decoding and memory limitations.
Read further material that breaks down LLM inference from scratch: tokenization, embeddings, attention, the separation of pre-processing and decoding, key/value caches and quantization.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore