A hands-on tutorial on LLM inference: build a mini-sglang from scratch, then read the real SGLang source.
Start from what inference really is, build a mini-sglang by hand, then read the real SGLang source
Datawhale × RadixArk
Course status: Part 0, Part I, and Chapters 1–2 of Part II are complete in both English and Chinese. The remaining chapters of Part II and Parts III–IV are under development. You can switch to 简体中文 at any time. Want to help? See Contributing.
Large models have moved on from the training race to the inference race. Once applications took off, inference cost and latency became the real bottleneck: on the same GPU, a good inference engine can serve several times more requests. Yet there are few tutorials on inference engines. They either stay at the concept level without touching code, or drop you straight into the SGLang source with no idea where to start.
This course fills that gap. The first half starts from the core questions of inference: why KV Cache exists, how prefill differs from decode, what compute-bound and memory-bound actually mean. The middle part walks you through building a mini-sglang from scratch: forward pass, generation, KV Cache, HTTP serving, Continuous Batching, Paged KV Cache, RadixAttention, added one at a time. The last part returns to the real SGLang, explains how its cutting-edge optimizations are actually implemented, and finally shows you how to land your first solid PR in SGLang.
The project is jointly launched by Datawhale and RadixArk (the company founded by the SGLang team). SGLang is technically solid and actively iterated in the inference framework space. If you like this project, please give the official SGLang repository a ⭐.
- 📖 Understand inference: KV Cache, prefill/decode, compute-bound vs. memory-bound, and what these concepts actually mean
- 🏗️ Build the engine