{
  "id": 7569687,
  "title": "One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy",
  "url": "https://urgent.news/2026/09/15/one-table-many-engines-running-bigquery-spark-and-trino-on-a-single",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-15T14:41:39.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/carlosrgomes/one-table-many-engines-running-bigquery-spark-and-trino-on-a-single-iceberg-copy-4b9m"
  },
  "original_language": "en",
  "account": "Data professionals often use different tools for various tasks. Business intelligence analysts utilize SQL within Google BigQuery to create sales reports. Data scientists employ Apache Spark to train forecasting models. Infrastructure engineers utilize Trino for rapid, on-the-fly queries. Traditionally, each team would copy the identical data into their respective platforms. This approach led to several issues: increased storage costs as the same data was billed multiple times, outdated information when one copy remained unchanged after a cancellation, and separate security configurations for each tool. The concept of a \"Data Lakehouse\" emerged to solve these problems. Lakehouse represents a unified platform that stores a single copy of data in affordable cloud storage (like Google Cloud Storage) and allows multiple query engines (SQL, Spark, Python, Trino) to access the data simultaneously using clear transaction rules. To understand how multiple engines like Google BigQuery, Apache Spark, and Trino can interact with the same data simultaneously, it's essential to comprehend the core components involved:\n\n1. Google Cloud Storage (GCS): Functions as a cloud-based hard drive where raw files (such as CSV, JSON, Parquet) are stored reliably and cost-effectively. In this setup, it houses both the actual data and the associated metadata files.\n\n2. The Parquet Format: Parquet is a method for saving table data to files. Unlike CSV, which stores information row by row, Parquet organizes data by column and applies extensive compression techniques. This results in faster data access and efficient querying.\n\n3. Apache Iceberg: Iceberg is an open metadata layer built upon Parquet files that transform them into actual tables. By doing so, it offers several benefits:\n- ACID transactions: Guarantees that either the entire data write operation succeeds, or none of it changes.\n- Time travel: Allows querying of the table's state at specific points in time, thanks to its ability to maintain a history of versions called snapshots.\n- Schema evolution: Enables adding or renaming columns without disrupting existing queries.\n\n4. Lakehouse for Apache Iceberg (formerly BigLake): Lakehouse is a Google Cloud service previously known as BigLake. It bridges the gap between file storage (Cloud Storage) and SQL analytics (BigQuery). This platform offers two significant advantages:\n- Native Iceberg REST Catalog: Implements the Iceberg standard for registering tables and managing current data versions. Lakehouse provides a central hub that facilitates read/write interoperability between BigQuery, Google Cloud Managed Service for Apache Spark, and other Iceberg-compatible open-source engines (Spark, Trino, Flink). Notably, it eliminates the need for data to be trapped in proprietary formats.\n- Unified security via Workload Identity: Instead of managing risky password files (like Service Account .json keys), Lakehouse leverages the cloud's own execution identity for authorization. This centralizes access permissions and reduces security risks.\n\nTo ensure that three different engines (BigQuery, Spark, and Trino) can access and modify the same data concurrently without conflicts, Optimistic Concurrency Control (OCC) is employed. The OCC process functions as follows:\n\n1. Initial read: Both BigQuery and Spark read the current table version from the Lakehouse catalog (e.g., snapshot number 10).\n2. Isolated work: Each application independently computes its changes and writes new files to Cloud Storage without being aware of the other's activities.\n3. Commit process: Once one application (e.g., BigQuery) finishes its changes, it requests Lakehouse to commit those changes over snapshot 10. If the current version remains the same (10), the write is successfully applied.\n\nThis concurrency control mechanism ensures that only one application can overwrite changes at any given time, preventing data conflicts and maintaining data integrity across all query engines.",
  "summary": "If you work with data — or you're just getting started in the cloud — you've probably heard terms like Data Warehouse, Data Lake, and Data Lakehouse. This article explains those concepts from scratch, details what Lakehouse for Apache Iceberg on Google Cloud (formerly BigLake ) and Apache Iceberg actually are, and shows how to make three different tools read and modify the same data at the same…",
  "key_points": [
    "Google Cloud Storage houses raw data and metadata files",
    "Parquet format compresses data column-wise for efficient querying",
    "Apache Iceberg adds ACID transactions, time travel, and schema evolution"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}