Delta Lake UniForm Is the Real Fix for the Snowflake-vs-Databricks Standoff

Delta Lake UniForm, also known as Universal Format, aims to address the ongoing debate between Snowflake and Databricks over which platform should own a dataset. While both tools have their strengths, ownership of the data format ultimately determines who has first-class access to the data. In most cases, the platform that doesn't obtain the workload loses out on having direct access to the data.

To counter this issue, Delta Lake UniForm enables organizations to write a table in Delta Lake format and automatically generate Iceberg (and Hudi) metadata alongside the Delta log, pointing to the same underlying Parquet files. This way, any engine that supports Delta, Iceberg, or Hudi can access the table natively. This single mechanism ensures that Databricks jobs continue writing Delta tables as usual, while Snowflake can read the same table as an Iceberg table without any export or additional storage costs.

The setup requires several steps, starting with enabling UniForm on a Delta table in Databricks and registering a service principal with access to Unity Catalog, which includes the EXTERNAL USE SCHEMA permission. The next step is to secure traffic using Managed Private Endpoints on Azure, allowing both Snowflake's calls to Unity Catalog's catalog API and its reads against storage to travel over Azure's private network fabric.

After configuring the private path, linking Snowflake to Unity Catalog as a catalog-linked database is the final step. This enables Snowflake to automatically access Unity Catalog's metadata, including Iceberg metadata generated by Delta Lake UniForm, eliminating the need to manually create Iceberg tables in Snowflake. Although the metadata pull is not real-time, it eliminates the need for per-table busywork.

However, it's important to note that writing back to Databricks from Snowflake is still a newer feature and direction-dependent.