data-processing-nemo-curator

Topics in the source

• NeMo Curator - GPU-Accelerated Data Curation
• When to use NeMo Curator
• Quick start
• Installation
• Basic text curation pipeline
• Data curation pipeline
• Stage 1: Quality filtering
• Stage 2: Deduplication

NVIDIA's toolkit for preparing high-quality training data for LLMs. Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi-modal datasets (text, images, video, audio) Filtering low-quality or toxic content Scaling data processing across GPU cluster Performance: 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near-linear scaling across GPU nodes Use alternatives instead: datatrove: CPU-based, open-source data processing dolma: Allen AI's data toolkit Ray Data: General ML data processing (no curation focus) Exact deduplication: Fuzzy deduplication (16× faster on GPU): Semantic deduplication: | Operation | CPU (16 cores) | GPU (A100) | Speedup | |-----------|----------------|------------|---------| | Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× | | Exact dedup (1TB) | 8 hours | 0.5 hours | 16× | | Quality filtering | 2 hours | 0.2 hours | 10× | CPU (256 cores): 120 hours GPU (8× A100): 7.5 hours Speedup: 16× CPU (64 cores): 8 hours GPU (4× A100): 0.5 hours Speedup: 16× CPU (32 cores): 2 hours GPU (2× A100): 0.2 hours Speedup: 10× CPU-based curation (AWS c5.18

This is an excerpt. Refer to the complete source and its current requirements.