Building an AWS + Snowflake Landing Zone with Terraform: Lessons from My CI/CD Review

Building a robust AWS + Snowflake landing zone with Terraform requires a thoughtful approach, as shared in a recent CI/CD review. The core of the platform is the raw/landing tier where source data initially lands. While this can be set up manually once, ensuring reproducibility across different environments like dev, UAT, and prod, and proving that the production environment mirrors the reviewed code, becomes significantly more challenging.

To address these challenges, the author built the entire landing tier of a banking-style data platform using Terraform. This included a single root module that orchestrated five child modules, with remote state management, environment promotion, and integrated into a GitLab pipeline that gates every change. The resulting code allowed for explicit dependencies, reusability across different stacks, and state that would not corrupt itself.

During the review of this Terraform code, the author identified three critical design gaps that highlight the importance of CI/CD in ensuring the safety and security of infrastructure-as-code.

Firstly, merge requests (MRs) did not produce any pipeline at all. Since MR events were killed with `when: never`, reviewers approved changes without seeing what the actual impact would be on the infrastructure. This oversight meant that reviewers were approving changes blindly, without evidence of the intended outcome. The solution was to introduce a dedicated MR job that runs `validate` and `plan` stages and posts the plan output on the merge request. This approach ensures that approval is based on evidence, rather than blind acceptance.

Secondly, the main branch automatically applied changes to the production environment without any gate. While the pipeline applied changes to production automatically when merging to the main branch, the `destroy` action was manual and blocked by merges to main. This asymmetry is counterproductive because teardown, which is inherently riskier, lacks the protective gate that the automatic apply to prod lacks.

The correct approach is to require manual approval for any production changes, ensuring that the potentially riskier operations are reviewed and approved explicitly.

Thirdly, secrets were hardcoded in *.tfvars files, and the `.gitignore` configuration inadvertently tracked these files. Storing credentials in version control is a significant security risk. The recommended practice is to use CI variables or a secrets manager for credentials. The variables should be marked as sensitive, and the *.tfvars files should be ignored in `.gitignore` to prevent tracking of these sensitive files.

Any exposed secrets must be rigorously rotated and any history containing these secrets must be scrubbed to maintain security.

These lessons emphasize that while Terraform makes it easy to provision and manage infrastructure, the pipeline designed around it is equally crucial for ensuring that the infrastructure is not only reproducible but also reviewable and secure. A well-structured Terraform configuration is a good foundation, but the CI/CD pipeline is what truly makes the infrastructure safe and reliable.

The full Terraform code, including five modules, GitLab CI pipeline, and environment variable files, is available at [this link](https://gitlab.com/kiran.gntdm-group/terraform) for those interested in exploring the implementation further.