{
  "id": 9749082,
  "title": "Fixing Checkpoint Recovery in SeaTunnel’s BigQuery Connector",
  "url": "https://urgent.news/2026/09/25/fixing-checkpoint-recovery-in-seatunnels-bigquery-connector",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-25T06:36:10.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/fixing-checkpoint-recovery-in-seatunnels-bigquery-connector?source=rss"
  },
  "original_language": "en",
  "account": "The BigQuery Sink Connector for Apache SeaTunnel initially aimed to write rows into BigQuery by using the BigQuery Storage Write API. However, the real challenge was aligning BigQuery's external write visibility model with SeaTunnel's checkpoint and restore lifecycle. The initial design used Pending Streams for batch writes, which seemed like a natural fit. The flow was as follows: create a pending stream, append rows, finalize the write stream, and then commit the write streams atomically through BatchCommitWriteStreams. This approach looked similar to a two-phase commit protocol, with prepareCommit() finalizing the stream and then calling finalizeStream() and BatchCommitWriteStreams(). However, during review, a subtle problem emerged. After a failed checkpoint, prepareCommit() had already finalized the stream by calling FinalizeWriteStream. This created an ownership gap during restore. The checkpoint state discarded the finalized stream, leading to the issue of failed-checkpoint streams not being part of the restored state. The solution was to redesign the batch write path to use Buffered Streams instead of Pending Streams. Buffered Streams did not commit the whole stream through a finalize-and-batch-commit protocol. Instead, they made data visible up to a specific stream offset. The new design had a writer state representing the stream name, next offset, and checkpoint ID. The prepareCommit() method flushed the stream and returned a BigQueryCommitInfo containing the stream name and flush offset. The snapshotState() method returned a BigQuerySinkState containing the stream name, next offset, and checkpoint ID. The commit() method flushed the rows up to a specific offset, and the restoreWriter() method selected the latest completed checkpoint state to restore the writer. The key takeaway was understanding the difference between writer state and commit info. Writer state represented where the writer should resume appending after restore, while commit info represented the external write position in BigQuery. This redesign allowed the connector to have a more explicit story about external side effects, ensuring clear checkpoint recovery semantics.",
  "summary": "Apache SeaTunnel improved the BigQuery Sink Connector with thoughtful ideas and innovation, making data integration easier and better for everyone!",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}