Airflow
Apache Airflow is an open-source workflow management platform for data engineering pipelines. It started at Airbnb in October 2014 as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and…
Key facts
- First appeared
- 2014
- Category
- technology
- Problem solved
- Apache Airflow was created to address the limitations of traditional cron jobs and simple scripting for orchestrating complex, interdependent data pipelines. It aimed to provide a scalable, reliable, and visible solution for defining, scheduling, and monitoring workflows, offering features like retries, backfilling, and a clear user interface that ad-hoc scripts lacked.
- Platforms
- web, kubernetes, linux, azure_cloud, gcp_cloud, macos, docker, aws_cloud
Related technologies
Notable users
- Robinhood
- Square
- Airbnb
- Adobe
- ING
- Spotify
- Lyft