Amazon EKS announced it now supports Kubernetes clusters with up to 100,000 nodes. The authors name massive AI/ML workloads as a possible use case for such setups, which can fit 1.6m AWS Trainium chips or 800k NVIDIA GPUs in a single K8s cluster.

It’s been a significant effort in AWS involving a comprehensive set of improvements aimed at achieving such a scale. Here’s what the engineers did:

- Re-architecting etcd (its new design is illustrated in this post). It involved switching from a Raft-based consensus backend to an internal component built at AWS, moving BoltDB from network-attached Amazon EBS volumes to in-memory storage (tmpfs), and choosing an optimal partitioning scheme.
- Tuning API servers by elaborating specific configurations, enabling strongly-consistent reads from cache and streaming list responses, and using CBOR (Concise Binary Object Representation) encoding for custom resources.
- Optimizing cluster controllers' performance, enhancing Karpenter, scaling the cluster network, and introducing SOCI (Seekable OCI) fast pull for container image pulls.

Find more details, including the resulting benchmark charts, in this blog post.

#news #AWS