data-processing-ray-data
Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.
Target: Learning & research. Automatically collected · unverified.
Excerpt from the author’s documentation
Extracted from a fixed version of SKILL.md in its original language. This is the author’s documentation; installation, compatibility and runtime results have not been verified by this site.
Read the original English text
Topics in the source
- Ray Data - Scalable ML Data Processing
- When to use Ray Data
- Quick start
- Installation
- Load and transform data
- Integration with Ray Train
- Reading data
- From cloud storage
Distributed data processing library for ML and AI workloads. Use Ray Data when: Processing large datasets (>100GB) for ML training Need distributed data preprocessing across cluster Building batch inference pipelines Loading multi-modal data (images, audio, video) Scaling data processing from laptop to cluster Key features: Streaming execution: Process data larger than memory GPU support: Accelerate transforms with GPUs Framework integration: PyTorch, TensorFlow, HuggingFace Multi-modal: Images, Parquet, CSV, JSON, audio, video Use alternatives instead: Pandas: Small data (<1GB) on single machine Dask: Tabular data, SQL-like operations Spark: Enterprise ETL, SQL queries | Format | Read | Write | Use Case | |--------|------|-------|----------| | Parquet | ✅ | ✅ | ML data (recommended) | | CSV | ✅ | ✅ | Tabular data | | JSON | ✅ | ✅ | Semi-structured | | Images | ✅ | ❌ | Computer vision | | NumPy | ✅ | ✅ | Arrays | | Pandas | ✅ | ❌ | DataFrames | Scaling (processing 100GB data): 1 node (16 cores): 30 minutes 4 nodes (64 cores): 8 minutes 16 nodes (256 cores): 2 minutes GPU acceleration (image preprocessing): CPU only: 1,000 images/sec 1 GPU: 5,000 images/sec 4 GPUs: 18,000 images/sec
This is an excerpt. Refer to the complete source and its current requirements.
- Source repository
- davila7/claude-code-templates
- Source file
- cli-tool/components/skills/ai-research/data-processing-ray-data/SKILL.md
- Fixed revision
- 21be197f100a12b8d0dd9a211cc244c8179f84d3
Sources and evidence
This record was collected automatically. Its documentation, suitability and runtime behavior have not been manually verified.
Author: davila7/claude-code-templates · original repository metadata. Documentation review: Not reviewed. Last source check: 2026-11-09T14:01:43.028Z.
Categories are inferred from names, descriptions and paths. They do not establish compatibility.
License and attribution
MIT. Third-party content and copyright remain with their authors. Refer to the original license and terms.
Original license ↗Synchronization, copying and user-reported success do not mark a resource as runtime-tested.
Limitations and notes
- Collecting metadata does not mean this site has installed, adapted, runtime-tested or security-reviewed the resource.