Databricks Architecture Explained: Control Plane, Compute Plane, Delta Lake and Unity Catalog
"Explain the Databricks architecture" is one of the most common data engineering questions. Here is the answer as a picture, built up layer by layer. 1. It starts with your cloud account Databricks runs on Azure, AWS or Google Cloud, and your cloud account is the foundation everything else sits on. 2. The control plane Databricks runs as two halves. The control plane is hosted and managed by…
Databricks architecture is a layered system consisting of several key components.
The foundation is your cloud account, which can be set up on Azure, AWS, or Google Cloud. This account serves as the base upon which Databricks is built.
Above the cloud account lies the control plane, which is managed and hosted by Databricks. This is where you will find the web application, notebooks, job scheduler, and cluster manager.
Beneath the control plane is the compute plane, also known as the data plane. This is where the actual computing power is executed. In most traditional setups, it resides within your own cloud account within your virtual network. However, with serverless compute, it operates within Databricks' cloud account instead.
At the bottom of the entire architecture is your data lake, which could be ADLS Gen2 on Azure or S3 on AWS. This is where all raw files are stored, and they never leave your own account.
On top of the lake sits Delta Lake, a storage layer that adds ACID transactions, schema enforcement, and time travel capabilities to plain Parquet files. This is what transforms a data lake into a lakehouse.
Delta Lake organizes data into three different zones: Bronze for raw ingested data, Silver for cleaned and joined data, and Gold for business-level aggregates ready for reporting.
Spanning across all these components is the Unity Catalog, the governance layer. It handles permissions, lineage, and provides a single catalog across every workspace.
When you run a notebook, the control plane sends the instruction. It triggers a cluster in the compute plane to read from Delta and write results back. However, only results and metadata are returned to the control plane, ensuring that the actual data remains untouched and secure throughout the process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
