Apache Arrow
Apache Arrow is an open-source, language-agnostic specification for in-memory columnar data. It provides a standardized memory format that enables zero-copy data exchange between different systems and programming languages, drastically reducing serialization/deserialization overhead and…
Key facts
- First appeared
- 2016
- Category
- technology
- Problem solved
- Apache Arrow was created to solve the pervasive problem of inefficient data exchange and processing in big data and data science ecosystems. Prior to Arrow, moving data between different programming languages (e.g., Python, Java, R, C++) or systems (e.g., Spark, Pandas, databases) required costly serialization, deserialization, and copying due to disparate in-memory data representations. This overhead created significant performance bottlenecks and wasted CPU cycles, especially for analytical workloads that frequently transfer large datasets.
- Platforms
- ARM, x86, Windows, Linux, macOS
Related technologies
Notable users
- Voltron Data
- Microsoft Azure
- Ursa Computing
- Google Cloud
- Dremio
- Amazon Web Services (AWS)
- NVIDIA
- Two Sigma
- Databricks