inference-serving-sglang

Topics in the source

• SGLang
• When to use SGLang
• Quick start
• Installation
• Launch server
• Basic inference
• Structured JSON output
• RadixAttention (Key Innovation)

High-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching. Use SGLang when: Need structured outputs (JSON, regex, grammar) Building agents with repeated prefixes (system prompts, tools) Agentic workflows with function calling Multi-turn conversations with shared context Need faster JSON decoding (3× vs standard) Use vLLM instead when: Simple text generation without structure Don't need prefix caching Want mature, widely-tested production system Use TensorRT-LLM instead when: Maximum single-request latency (no batching needed) NVIDIA-only deployment Need FP8/INT4 quantization on H100 What it does: Automatically caches and reuses common prefixes across requests. Performance: 5× faster for agentic workloads with shared system prompts 10× faster for few-shot prompting with repeated examples Zero configuration - works automatically How it works: 1. Builds radix tree of all processed tokens 2. Automatically detects shared prefixes 3. Reuses KV cache for matching prefixes 4. Only computes new tokens Example (Agent with system prompt): Few-shot prompting (10 examples in prompt): vLLM: 2.5 sec/request SGLang: 0.25 sec/request (10× faster) Throughp

This is an excerpt. Refer to the complete source and its current requirements.