Design review: self-hosted LLM API for a few isolated business tenants

I’m planning a small self-hosted inference service using open-weight models on a GPU server in a German data center. It would serve a few business tenants through an API. I’m working through tenant separation, per-customer API keys and quotas, prompt retention, logs, model updates, and recovery if the host or a GPU fails.

For people who have built self-hosted inference services, what architecture choices caused trouble once multiple tenants started using the service? Would you isolate tenants by separate model processes, containers, or physical GPUs? What should be handled by the gateway versus the inference server? I’m at the design stage, so concrete deployment lessons and security pitfalls would be especially helpful.

https://redd.it/1x3jhu4
@r_SelfHosted