Marquez

Marquez is an open source metadata service for collecting, aggregating, and visualizing a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, and gives global visibility into job runtime and how often datasets are accessed. It was released and open sourced by WeWork.

The service collects dataset, job, and run metadata using OpenLineage, of which it is the reference implementation. A core API stores metadata in PostgreSQL, a web UI shows dependencies between jobs and the datasets they produce and consume through a lineage graph, and HTTP and GraphQL endpoints are available. By default the HTTP API requires no authentication or authorization.

Marquez is an LF AI & Data Foundation Graduated project. The easiest way to start is with Docker, and it can also be built from source with Java 17 and PostgreSQL 14. Helm charts are provided for Kubernetes deployments.