DataFusion
Apache DataFusion is an extensible, Rust-native query engine library designed for high-performance in-memory and on-disk data processing. It provides a SQL query planner and execution engine that can be embedded into other applications, enabling efficient analytics directly on Apache Arrow data.…
Key facts
- First appeared
- 2019
- Category
- technology
- Problem solved
- DataFusion was created to provide a high-performance, in-memory, columnar SQL query engine that natively processes Apache Arrow data. This minimizes the common serialization/deserialization overhead prevalent in traditional data processing engines and enables efficient analytical workloads within Rust applications or integrated data platforms, offering a modular and extensible alternative to monolithic solutions.
- Platforms
- Any platform supported by Rust, macOS, Windows, Linux
Related technologies
Notable users
- DataBend (for DataBend/CeresDB)
- Open-source projects in the Apache Arrow ecosystem
- InfluxData (for InfluxDB IOx)
- Companies building custom Rust-based data platforms