How I stopped cloning datasets for Vertex AI pipelines
You ship the job. It passes CI. Then, the SRE team pings you on Slack because your pipeline just swallowed 400GB of storage in an hour, and your cost projection for the quarter looks like a down payment on a house. You’re duplicating data again. Most of us treat the Lakehouse as a write-once repository. We ETL, we transform, and then we export the "features" into a brand-new table or GCS bucket…
In this article, the author explains how to avoid data duplication when using Vertex AI pipelines. By treating your existing warehouse, such as BigQuery or Delta Lake, as the source of truth and connecting Vertex AI Pipelines directly to the raw or processed tables, you can prevent unnecessary storage costs and compliance issues.
The key is to let your feature tables stay in the warehouse and use BigQuery as the source for Vertex AI Pipelines. The author demonstrates how to implement this pattern using Google Cloud AI Platform SDK, specifically with KFP components that execute SQL queries directly on the existing lakehouse tables. This approach eliminates the need for materializing data into CSV files or creating new tables.
However, the author also highlights some trade-offs to consider. Performance may be non-deterministic when running complex queries during the training loop, and training on live data can introduce non-deterministic bias if upstream pipelines update feature tables mid-run. Additionally, there are challenges with Vertex AI SDK versioning, and the approach may not be suitable for real-time inference or when data scientists are still exploring feature designs.
The author recommends using this pattern when feature engineering is idempotent and BigQuery is already set up as the primary source of truth, such as in environments using dbt for data management. In cases requiring low-latency feature retrieval or when constant schema changes occur, the author suggests considering dedicated feature stores like Feast or Vertex AI Feature Store.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
