In Kubernetes clusters, GPUs are expensive. Really expensive. And when your data science team is fighting over GPU resources like it’s Black Friday at Best Buy, you know something’s broken.

That was the challenge I faced a few years ago. I implemented this solution back then, and though it’s been a while, I thought now would be a great time to share the experience.

We had a k8s cluster with 8 NVIDIA A100 GPUs, and our data scientists were… unhappy. Small model training jobs waited hours for GPU access while large distributed training jobs hogged entire GPUs at 30% utilization.

The culprit? GPU allocation granularity. Kubernetes treats GPUs as atomic resources, you either get a whole GPU or nothing. There’s no middle ground.

I needed to fix this. And the solution wasn’t what I expected.