ELT with Dataform on Google Cloud
This article was originally published on Medium (Google Cloud Community) . 1. Explanation of the use case presented in this article In a previous article, I demonstrated how to build an ELT pipeline using dbt on Google Cloud . In this article, we'll explore the same use case — this time using Dataform , Google Cloud's native solution for managing SQL-based data workflows. One of the key…
This article discusses an ELT pipeline implemented using Dataform on Google Cloud. In a previous article, the author demonstrated building an ELT pipeline using dbt on Google Cloud. This article focuses on the same use case but utilizes Dataform, which is a fully managed, serverless service natively integrated with BigQuery. The article provides a step-by-step guide on creating a Dataform repository manually, linking it with a GitHub repository, and automating the setup using Terraform, enabling a GitOps workflow.
The use case involves a football-related ELT pipeline with both staging and mart layers, including dynamic views and tables generation. The CI/CD layer is implemented using Cloud Build and Infrastructure as Code (IaC) with Terraform, automating the creation of the Dataform repository and linking it with a GitHub repository. The applicative ELT pipeline includes raw file loading into BigQuery, staging layer for initial data cleaning and standardization, mart layer for transforming cleaned data into business-ready domain models, and dynamic views and tables generation.
The Dataform project structure is explained, including the creation of a repository, development workspace, and workspace initialization. The workflow_settings.yaml file is opened, dependencies are installed, and the boilerplate project provides ready-to-run code. The pipeline can be executed directly in the UI, and the execution details can be viewed after completion.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
