Absolutely Everything You Always Wanted to Know About Watermarks in Apache Flink

Watermarks are a crucial aspect of Apache Flink, often discussed and sometimes challenging to grasp fully. This deep-dive article aims to provide an exhaustive explanation of watermarks in Flink, specifically for the Flink SQL and Table API. The scope of this article excludes basic concepts like event-time vs processing-time and does not cover Confluent Cloud's extensions for watermarks.

Watermarks are signals injected periodically by Flink into the dataflow, representing timestamps in milliseconds since the Epoch. These signals flow with data along the Dataflow, allowing each Subtask to keep track of its own Watermark time independently. The Watermark time of a Subtask determines the propagation downstream, ultimately reaching the Sinks.

At the Operator level (subtask level), Watermarks are generated, propagated, and used to determine when time-based operations, such as time windows or temporal joins, should trigger. For instance, a time window ending at time T is closed when a Watermark with time ≥ T is observed by the Operator performing the window aggregation.

Watermarks are generated, propagated, and used at the Subtask level, with each Subtask keeping track of its Watermark time independently. A Subtask uses the minimum timestamp across all its active inputs to determine its Watermark time. For Source Subtasks, this time is the maximum (newest) event-time observed in that Kafka partition, while for other Subtasks, it is the Watermark received on that input channel.

Events with an event-time older than or equal to the Subtask's Watermark are considered late events and are silently dropped by most time-based operations, such as window aggregations. However, some time-based operators, like Interval Joins, handle late events differently by dropping them only when time-based operations are involved.

In summary, watermarks play a vital role in Apache Flink, ensuring that time-based operations can function correctly by tracking and propagating timestamps across the dataflow. This comprehensive article on watermarks in Flink provides a solid foundation for understanding these essential signals and their behavior within the Flink ecosystem.