inference-serving-tensorrt-llm

Topics in the source

• TensorRT-LLM
• When to use TensorRT-LLM
• Quick start
• Installation
• Basic inference
• Serving with trtllm-serve
• Key features
• Performance optimizations

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs. Use TensorRT-LLM when: Deploying on NVIDIA GPUs (A100, H100, GB200) Need maximum throughput (24,000+ tokens/sec on Llama 3) Require low latency for real-time applications Working with quantized models (FP8, INT4, FP4) Scaling across multiple GPUs or nodes Use vLLM instead when: Need simpler setup and Python-first API Want PagedAttention without TensorRT compilation Working with AMD GPUs or non-NVIDIA hardware Use llama.cpp instead when: Deploying on CPU or Apple Silicon Need edge deployment without NVIDIA GPUs Want simpler GGUF quantization format In-flight batching: Dynamic batching during generation Paged KV cache: Efficient memory management Flash Attention: Optimized attention kernels Quantization: FP8, INT4, FP4 for 2-4× faster inference CUDA graphs: Reduced kernel launch overhead Tensor parallelism (TP): Split model across GPUs Pipeline parallelism (PP): Layer-wise distribution Expert parallelism: For Mixture-of-Experts models Multi-node: Scale beyond single machine Speculative decoding: Faster generation with draft models LoRA serving: Efficient multi-adapter deploymen

This is an excerpt. Refer to the complete source and its current requirements.