Apache Airflow

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows as code. It works best with pipelines that are mostly static and slowly changing, and is widely used for data processing and orchestration.

Workflows are defined as DAGs that orchestrate tasks. The scheduler executes those tasks across an array of workers while following the dependencies you specify, and the web UI visualizes running pipelines, tracks progress, and helps troubleshoot failures. Pipelines are defined in code, so DAGs can be generated and parameterized dynamically, and Jinja templating allows rich customization. A wide range of built-in operators ships with the framework and can be extended.

Airflow runs on POSIX-compliant operating systems, with Linux as the recommended production environment and macOS supported for development. On Windows it runs through WSL2 or Linux containers. Releases are distributed via PyPI and official Docker images.