Apache Airflow
airflow.apache.orgProgrammatically author, schedule and monitor workflows with Python
Data & Analyticsworkflow-orchestrationdata-pipelinespythonopen-sourceetlml-opscloud-integration
About
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data and ML workflows. Pipelines are defined in Python, enabling dynamic generation and full flexibility. It features a modern web UI, scalable architecture, and integrations with major cloud providers like AWS, GCP, and Azure.
Problem
Managing complex, scheduled workflows across distributed infrastructure is difficult without a centralized, code-driven orchestration tool.
For
data engineers and developers building and managing automated workflows
How it works
Users define workflows as Python DAGs (Directed Acyclic Graphs), which Airflow schedules, executes via workers, and monitors through a web interface.
Business model
open-source
Status
launched
Company
Apache Software Foundation