Without a graphics processor. Without PyTorch. Without inference loops on the central processor.
Just a transformer, embedded in hardware, generating 50,000+ tokens per second.
The model is small, but the point is not that. The point is that inference does not have to exist only in a software environment.
The goal was not to create the largest possible model.
The goal was to present the entire path of transformer inference in a form readable for hardware: memory, counters, state machines, accumulators, lookup tables, and multi-cycle arithmetic blocks.
The base scheme uses fixed Q4.12 arithmetic and weights stored in ROM.
Most of the model boils down to one repetitive operation: matrix-vector multiplication. Therefore, a reusable 16-channel stream block for matrix-vector calculations was implemented, and then it was temporarily multiplexed to Q/K/V, MLP, and the output layer of the language model.
The most interesting was the attention mechanism.
In Python, it's a single neat equation.
In RTL, it turns into a schedule: generation of Q/K/V, passage through scalar products, tracking the maximum, approximate calculation of the exponential, accumulation, division, mixing V, then reverse projection.
Source
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore