LLM break all the assumptions on which classical inference in ML was built.

Traditional model (convolutional neural network, transformer classifier, XGBoost) outputs a result in a single forward pass.

Nothing is stored between requests. The graphics processor performs the same type of work each time.

➡️ How LLM work completely differently?

The output is generated one token at a time, autoregressively, which turns a single request into hundreds of consecutive forward passes.

The prefill stage relies on computations, and decoding relies on memory bandwidth, and running them together on a single graphics processor degrades both stages.

The key-value cache grows with the length of the dialogue and is shared between requests, so routing is no longer about the least loaded servers, but about which replica already has the needed prefix cached.

Models with a mix of experts add parallelism of experts on top of this.

None of this is in classical machine learning model serving.

That's why a separate stack of optimizations specifically for inference of large language models has emerged: compression, attention mechanism, key-value cache management, batching, decoding, parallelism, and routing.

The demo shows a photo map of 72 techniques for optimizing large language models in production, grouped into nine blocks.

I also attach an article that explains how inference of large language models differs from classical inference and why each of these blocks is needed.

Question: what other techniques for optimizing large language models would you add here?

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore