​​How Do You Fit a Trillion-Parameter Model Into a Kubernetes Cluster? is an interesting article about how one should change their perspective when reasoning about running LLMs in Kubernetes compared to usual web apps.

It’s an interesting read, especially, if you don’t work with this stuff every day. The biggest takeaway here is that in the case of models, a “replica” doesn’t mean a pod in most of the cases, it’s a distributed system on its own that should behave as one. This article also explains, how exactly things are distributed within a replica, and what are the low level system parameters to pay attention to.

#kubernetes #ai #llm