vaex

Topics in the source

• Vaex
• Overview
• When to Use This Skill
• Core Capabilities
• 1. DataFrames and Data Loading
• 2. Data Processing and Manipulation
• 3. Performance and Optimization
• 4. Data Visualization

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows. Use Vaex when: Processing tabular datasets larger than available RAM (gigabytes to terabytes) Performing fast statistical aggregations on massive datasets Creating visualizations and heatmaps of large datasets Building machine learning pipelines on big data Converting between data formats (CSV, HDF5, Arrow, Parquet) Needing lazy evaluation and virtual columns to avoid memory overhead Working with astronomical data, financial time series, or other large-scale scientific datasets Vaex provides six primary capability areas, each documented in detail in the references directory: Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference references/coredataframes.md for: Opening large files efficiently Converting from pandas/NumPy/Arrow Working with example datasets Understanding DataFrame structu

This is an excerpt. Refer to the complete source and its current requirements.