This article walks through a production deployment that solves all of the above using llm-d for intelligent inference scheduling, Istio as a Gateway API provider with Inference Extension support, and LiteLLM as a unified API gateway.
https://medium.com/@prasannanattuthurai/serving-multiple-llms-on-kubernetes-with-intelligent-routing-using-llm-d-istio-and-litellm-7d33760d1001