Auto Loader and Structured Streaming on Azure Databricks: Checkpoints, Schema Evolution, and Exactly-Once
Almost every Azure Databricks project I've seen starts the same way: files land in ADLS Gen2, and something has to pick up only the new ones, reliably, without reprocessing yesterday's data. Auto Loader plus Structured Streaming is the standard answer, and it's also a favourite interview topic because the details (checkpoints, schema evolution, exactly-once) separate people who've run it in…
Auto Loader and Structured Streaming on Azure Databricks offer a standard solution for reliably processing new files from Azure Data Lake Storage Gen2 (ADLS Gen2) without reprocessing older data. This approach is particularly useful for batch-style ingestion, where the notebook can be scheduled as a Databricks job to run at regular intervals.
The process begins with the creation of a minimal Auto Loader pipeline in PySpark, where the source path points to the ADLS Gen2 location containing the files. Checkpoints are set up at a separate location to track which files have already been ingested, preventing duplicate processing. Raw data is read in as JSON and schema is inferred, converting columns to their appropriate data types.
Two key aspects to consider are the trigger mechanism and checkpoint management. The trigger can be set to either process everything available and then stop (availableNow=True) or run continuously with a specified processing time interval (e.g., every 5 minutes). AvailableNow is preferable for most batch-style ingestion scenarios due to its ability to respect rate limits and prevent memory blow-ups.
Checkpoints store source offsets, batch commitment status, and information about discovered files. They must remain unique per stream to avoid corruption and should be maintained consistently with the data lifecycle - newly recreated tables should have associated checkpoints. Additional details such as the file path for lineage and schema evolution modes (adding new columns without breaking the stream) are also crucial for effective data management.
Schema evolution allows for seamless handling of additional columns added by upstream teams. The default mode (addNewColumns) requires a restart of the stream when new columns appear, after which the schema is updated and future runs include these columns. Other modes, such as failOnNewColumns, would cause the stream to fail until a schema change is made. Delta tables can be adapted to accept new columns by using .option(mergeSchema,true), ensuring schema flexibility without disrupting the pipeline.
Data quality checks can be implemented by filtering for rows marked as rescued data (_rescued_data IS NOT NULL), providing insights into how well the pipeline adapts to schema changes and unexpected data. Overall, these practices ensure that Azure Databricks projects can process large volumes of data efficiently and adapt to evolving data structures without extensive reprocessing or downtime.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
