Apache Paimon

Apache Paimon is an open lake format for building a real-time lakehouse on Flink and Spark. It handles both streaming and batch operations on the same tables, so a single dataset can serve continuous pipelines and periodic jobs alike.

Paimon pairs a lake table format with an LSM tree structure, which lets it absorb high-frequency updates and deletes and bring real-time streaming changes directly into the lake. Engines read and write these tables without moving the data into a separate store.

Paimon started as Flink Table Store in the Flink community before becoming its own project. Its design takes cues from Iceberg.