COURSE 17 OF 22
Reliable Batch and Incremental Pipelines
Load changes safely, retry work, and recover without duplicating results.
Background reading
Introduction to Reliable Batch and Incremental Pipelines
Yesterday’s deliveries are already in the warehouse. This morning, a new export includes a new delivery and a correction to an old one. Appending every row would duplicate history. Reloading everything may be wasteful.
An incremental load processes a bounded set of changes. Its correctness depends on identifying those changes and applying them safely, including when the same batch runs twice.
Loading course…