GCP Dataflow

GCP Dataflow is a fully managed service on Google Cloud Platform that executes data processing pipelines, based on the open-source Apache Beam unified programming model. It provides serverless, fast, and cost-effective processing for both batch and streaming data workloads, abstracting away…

Key facts

First appeared: 2014
Category: technology
Problem solved: GCP Dataflow was created to solve the fundamental problem of building and operating complex, large-scale data processing pipelines that could reliably handle both historical (batch) and real-time (streaming) data with a single, unified programming model, alleviating the operational burden of managing distributed infrastructure.
Platforms: Google Cloud Platform

Related technologies

Apache Kafka
Google BigQuery
Google Cloud Composer
Dataproc (for hybrid scenarios)
Google Cloud Pub/Sub
Google Cloud Spanner
Google Cloud Storage
Google Cloud Bigtable

Notable users

Google (internal use cases for services like Search, Ads, YouTube)
The New York Times
Sky
Activision Blizzard
AT&T
King