model-serving

vllm

+455

A high-throughput and memory-efficient inference and serving engine for LLMs

sglang-omni

+166

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

vllm-omni

+84

A framework for efficient model inference with omni-modality models

vllm-ascend

+49

Community maintained hardware plugin for vLLM on Huawei Ascend

flyteorg/flyte

+47

Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)

kserve/kserve

+25

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

BentoML

+14

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

paddock

+9

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

truss

+5

The simplest way to serve AI/ML models in production