Managing GenAI workloads on Kubernetes is surely gaining momentum. If it piques your interest as well, consider this new Open Source project.

LLMariner is an extensible platform for hosting and managing LLMs on K8s. It consists of a control plane and worker planes, which can be run in a single or across multiple Kubernetes clusters. Some of the project’s highlights are:

- Support for various inference runtimes, including vLLM, Ollama, and Triton.
- Support for numerous models (Llama 3.1, Gemma, TinyLlama, DeepSeek Coder, Mistral, and more), as well as other models via HuggingFace.
- Works with Retrieval-Augmented Generation (RAG).
- Allows to fine-tune models, run general-purpose training jobs, and run Jupyter Notebooks.
- User management via Dex and access control via organizations and projects.
- Integrates with Open WebUI and other tooling via OpenAI-compatible APIs.

Language: Go | License: Apache 2.0 | 53 ⭐️

▶️ GitHub repo
📢 Reddit announcement

#tools #aiml