If you’re interested in running local/private LLMs (leveraging Ollama and similar solutions) on Kubernetes, take a look at this new project.

KubeAI serves an OpenAI-compatible HTTP API in Kubernetes, providing you with a drop-in OpenAI replacement and simplifying the needed operations. The project’s authors call it “a Model Operator that manages vLLM and Ollama servers [inside Kubernetes].” Most noticeable KubeAI features include:

- Support for various Open Source model servers, including vLLM, Ollama, FasterWhisper, and Infinity. (Speech-to-Text and Text-Embedding are supported.)
- An option to preload LLMs in custom container images.
- Autoscaling based on load.
- A Chat UI based on OpenWebUI.
- An ability to work in the CPU-only mode and with GPUs. TPU support is planned.
- No dependencies (such as service meshes) and installable in regular K8s clusters, OpenShift, and managed K8s solutions (currently, there’s an instruction for GKE with Autopilot).

▶️ GitHub repo

#tools #aiml