#python

Mini-SGLang is a compact, easy-to-read inference framework (~5,000 Python lines) that runs and serves large language models with high speed using optimizations like radix cache, chunked prefill, overlap scheduling, tensor parallelism, and FlashAttention/FlashInfer kernels. It’s CUDA-dependent, quick to install from source, and can launch an OpenAI-compatible API or interactive shell for single- or multi‑GPU serving, letting you test or deploy models (e.g., Qwen, Llama) with low latency and scalable throughput. Benefit: you get a transparent, modifiable engine to deploy fast, efficient LLM inference for development, benchmarking, or production use.

https://github.com/sgl-project/mini-sglang