What self-hosted inference server actually shares one GPU across models?

We run three small models for our RAG setup, embeddings, a reranker, and a small extraction model, and each one has been sitting on its own box provisioned for peak load.

Last week I finally stopped guessing and actually measured the utilisation across them, and it came in low single digits, which means we're paying for these cards around the clock while they sit idle for most of the day.

The frustrating thing is that one model per server is just the default everyone accepts, and nobody questions it until the bill lands on your desk. So I've been hunting for anything that can keep more than one model on a single card, and here's where I've got to:


* Superlinked's SIE: keeps models hot on one GPU and evicts the least recently used when VRAM fills up, and it handles the extraction model too (I follow the project, so take it with salt)
* TEI (HF): rock solid, but it's one model per server, so three models means three separate boxes
* vLLM: brilliant for the big LLMs, but complete overkill for a handful of small encoders
* Infinity: serves embeddings and reranking from one process, which is handy if that's all you're running
* Baseten and the like: fully managed, which is fine unless the entire point was keeping things inside your own cloud

The one thing I still haven't tested properly is the cold start, because when a model gets evicted and has to reload in the middle of live traffic I have no real idea how badly the p99 suffers, and that's the number that decides whether this is production-ready for us.

Is anyone actually running multi-model on shared cards in production? I'd love to know how your eviction behaviour holds up once things get busy.

https://redd.it/1x5hfs9
@r_SelfHosted