data-processing-ray-data

Topics in the source

• Ray Data - Scalable ML Data Processing
• When to use Ray Data
• Quick start
• Installation
• Load and transform data
• Integration with Ray Train
• Reading data
• From cloud storage

Distributed data processing library for ML and AI workloads. Use Ray Data when: Processing large datasets (>100GB) for ML training Need distributed data preprocessing across cluster Building batch inference pipelines Loading multi-modal data (images, audio, video) Scaling data processing from laptop to cluster Key features: Streaming execution: Process data larger than memory GPU support: Accelerate transforms with GPUs Framework integration: PyTorch, TensorFlow, HuggingFace Multi-modal: Images, Parquet, CSV, JSON, audio, video Use alternatives instead: Pandas: Small data (<1GB) on single machine Dask: Tabular data, SQL-like operations Spark: Enterprise ETL, SQL queries | Format | Read | Write | Use Case | |--------|------|-------|----------| | Parquet | ✅ | ✅ | ML data (recommended) | | CSV | ✅ | ✅ | Tabular data | | JSON | ✅ | ✅ | Semi-structured | | Images | ✅ | ❌ | Computer vision | | NumPy | ✅ | ✅ | Arrays | | Pandas | ✅ | ❌ | DataFrames | Scaling (processing 100GB data): 1 node (16 cores): 30 minutes 4 nodes (64 cores): 8 minutes 16 nodes (256 cores): 2 minutes GPU acceleration (image preprocessing): CPU only: 1,000 images/sec 1 GPU: 5,000 images/sec 4 GPUs: 18,000 images/sec

This is an excerpt. Refer to the complete source and its current requirements.