inference-serving-llama-cpp

Topics in the source

• llama.cpp
• When to use llama.cpp
• Quick start
• Installation
• Download model
• Run inference
• Server mode
• Quantization formats

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware. Use llama.cpp when: Running on CPU-only machines Deploying on Apple Silicon (M1/M2/M3/M4) Using AMD or Intel GPUs (no CUDA) Edge deployment (Raspberry Pi, embedded systems) Need simple deployment without Docker/Python Use TensorRT-LLM instead when: Have NVIDIA GPUs (A100/H100) Need maximum throughput (100K+ tok/s) Running in datacenter with CUDA Use vLLM instead when: Have NVIDIA GPUs Need Python-first API Want PagedAttention | Format | Bits | Size (7B) | Speed | Quality | Use Case | |--------|------|-----------|-------|---------|----------| | Q4KM | 4.5 | 4.1 GB | Fast | Good | Recommended default | | Q4KS | 4.3 | 3.9 GB | Faster | Lower | Speed critical | | Q5KM | 5.5 | 4.8 GB | Medium | Better | Quality critical | | Q6K | 6.5 | 5.5 GB | Slower | Best | Maximum quality | | Q80 | 8.0 | 7.0 GB | Slow | Excellent | Minimal degradation | | Q2K | 2.5 | 2.7 GB | Fastest | Poor | Testing only | | CPU | Threads | Speed | Cost | |-----|---------|-------|------| | Apple M3 Max | 16 | 50 tok/s | $0 (local) | | AMD Ryzen 9 7950X | 32 | 35 tok/s | $0.50/hour | | Intel i9-13900K | 32 | 30 tok/s | $0

This is an excerpt. Refer to the complete source and its current requirements.