A future revision should not change yesterday's data selection
An as-of join can run successfully and still answer the wrong historical question. The important timestamp is when a particular version became available, not just the period the value describes. Suppose a synthetic September fact is published on October 3 with value 100, then revised on October 7 to 80. For an October 4 decision, the eligible value is 100. A pipeline that joins today's latest…
A synthetic September fact published on October 3 with a value of 100 was then revised on October 7 to a value of 80. For a decision made on October 4, the eligible value is 100. However, a pipeline that joins today's latest revision to every earlier date may incorrectly return 80 instead of the correct value. The important point is that adding a future revision must not change a past selection.
To ensure this, the availability of a value must be determined based on when that version became available, not just the period the value describes. The code provided demonstrates this concept with synthetic examples and asserts that the select function returns the correct value for different decisions made on October 4, October 8, and October 1.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
