Apache Parquet
Apache Parquet is an open-source columnar storage file format designed for efficient data storage and retrieval in big data processing frameworks like Apache Hadoop and Apache Spark. It optimizes analytical queries by storing data in a column-oriented fashion, enabling superior compression and…
Key facts
- First appeared
- 2013
- Category
- technology
- Problem solved
- Apache Parquet was created to address the inefficiencies of row-oriented data formats (like CSV, JSON, or Apache Avro) when performing analytical queries on massive datasets within distributed computing environments. Traditional formats required reading entire rows from disk, even if a query only needed a few columns, leading to excessive I/O, slow query times, and poor resource utilization. Parquet solved this by storing data column-by-column, allowing query engines to read only the necessary columns, significantly speeding up analytical workloads and reducing storage costs.
- Platforms
- Distributed File Systems, Google Cloud Storage, Azure Data Lake Storage, Local Filesystems, Apache Hadoop (HDFS), Amazon S3
Related technologies
Notable users
- Microsoft Azure
- Uber
- Cloudera
- Netflix
- Meta (Facebook)
- Amazon Web Services (AWS)
- Google Cloud Platform (GCP)
- Spotify
- Databricks