LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM

A quick recap of what we are doing on the project right now: we have a separate hardware server (hostname == “Matrix”), where we run our own self-hosted models with llama.cpp. In our Kubernetes cluster we have a LiteLLM AI Gateway for our clients – Backend API and other project services. Clients send their OpenAI/Anthropic/OpenRouter…

https://rtfm.co.ua/en/litellm-custom-callbacks-and-llm-evaluations-with-judge-llm/

#AI #LiteLLM #LLM #observability #Python #VictoriaMetrics #VictoriaTraces