data-pipelines

From the author

Build and repair batch and streaming pipelines that can be run twice without lying — lineage, event-time windowing, late arrivals, re-runnable writes and bounded backfills. Use when writing or fixing a pipeline, scheduled transform or job that produces data someone reads, when a run dropped or duplicated rows, or when planning a backfill. Not for designing the destination schema or its migrations, not for the standing assertions that guard the data afterwards (data-quality), and not for interpreting what the numbers mean.