Freshness Watermarks Belong on Every Row, Not the README

The author advocates for adding individual timestamps to every row in a dataset, rather than relying on a single banner timestamp at the top, such as a README or dashboard header. They argue that one banner timestamp can only accurately represent one portion of the dataset, while the rest may be misleading. By assigning a timestamp to each row, the author claims that it allows for more accurate tracking of when each piece of data was observed and ingested, and enables the detection of stale or outdated rows.

The author also notes that this approach helps to identify rows that have been tampered with through a process called "replay laundering," where an ETL process erroneously sets the as_of timestamp to the current time during a data replay, masking the true age of the data. The author provides a method for identifying and flagging these potentially problematic rows, which can be run per row, as opposed to the aggregate approach that the README banner design would require.

They emphasize that this per-row watermark design provides a clear and honest way to communicate the freshness of each data point, allowing consumers to easily see which rows are current and which may be outdated. The author concludes by stating that freshness should be treated as a property of each individual row, rather than an aggregate property of the entire dataset, and that every row should carry its own age information.