Apache Beam
Apache Beam is an open-source, unified programming model for defining and executing data processing pipelines, encompassing both batch and streaming patterns. It provides a portable API layer that allows developers to create data transformation jobs using various SDKs (Java, Python, Go) and…
Key facts
- First appeared
- 2016
- Category
- technology
- Problem solved
- Apache Beam was created to solve the fundamental problem of unifying batch and stream data processing, which previously required engineers to learn and implement separate, often inconsistent, frameworks for each paradigm. This led to code duplication, semantic inconsistencies (especially regarding event time and late data), operational complexity, and difficulty in ensuring correctness across different data processing modes. Beam offers a single programming model that abstracts away the underlying execution engine, allowing pipelines to be written once and run anywhere.
- Platforms
- Cloud (Google Cloud, AWS, Microsoft Azure), DirectRunner (local execution), Apache Flink, Apache Spark, On-premises (YARN, Kubernetes)
Related technologies
Notable users
- Spotify
- Microsoft (via Azure Data Factory/Synapse, sometimes using Beam concepts)
- PayPal
- T-Mobile
- Netflix
- Google (via Cloud Dataflow)