Prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python. With a few lines of code it turns ordinary scripts into production workflows, giving data teams scheduling, caching, retries, and event-based automations.

Flows can run on a schedule, be deployed, and trigger in response to events, and they handle retries, dependencies, and complex branching logic. Workflow activity is tracked so you can monitor it in a self-hosted Prefect server or the managed Prefect Cloud dashboard. A local server starts from the command line and serves its UI at localhost:4200.

Prefect is an open-source core with an optional managed cloud. It requires Python 3.10+, and a lightweight client package handles communication with a remote server or Prefect Cloud. Self-hosting gives you the full orchestration engine without a cloud account.