Building a Configuration-Driven Databricks Platform on AWS (Part 1)

Scaling bottlenecks in platform teams often stem from manual AWS console actions and scattered Terraform scripts whenever a new Databricks workspace is requested. This process involves choosing the appropriate AWS network, storage configuration, IAM settings, access assignments, compute options, and ensuring consistency across subsequent workspaces.

Terraform can define these resources, but the real challenge lies in managing the request process, transforming it into Terraform inputs, and deploying those changes to AWS and Databricks. This article outlines a configuration-driven approach utilizing the Databricks Platform Engineering repository. The subsequent part will detail setting up the development deployment from scratch.

The repository contains working development automation, along with infrastructure and account provisioning tailored for staging environments. The production files serve as configuration blueprints. This distinction is crucial when determining which components to leverage.

Platform teams frequently face bottlenecks due to manual AWS console clicks and fragmented Terraform scripts whenever a new Databricks workspace is requested. This request encompasses numerous decisions, including the selection of AWS account and region, network addresses, workspace storage bucket, access groups, and available compute policies and SQL warehouses.

These decisions can be documented in reviewed configuration files. In this implementation, each deployment is assigned two identifiers: version and environment. The version is set to 2, while the environment can be named 'dev' or 'sandbox'. The workspace name, such as 'demo', becomes the final cloud workspace name, 'dev-demo'. This naming convention ensures distinct workspace identification, such as 'dev-sandbox' or 'staging/shared'.

A filename alone does not define this deployment identity; additional workspaces must define another deployment within defined boundaries and execute the same provisioning procedures. Configuration should have a clear owner to streamline governance. Platform administrators are responsible for environment definitions, approved compute settings, personas, and access boundaries.

Team members request specific personas and data access through 'workloads/'. They do not dictate arbitrary infrastructure settings or create their own compute policies during these requests. For instance, the platform could define an 'analyst' persona with SQL access. A team could request this persona without selecting the SQL warehouse size or having unrestricted cluster creation capabilities.

Some settings are managed via supplemental JSON under 'platform/terraform-inputs/'. These inputs are strictly controlled and cannot override the settings defined in environment YAML files. This approach allows gradual expansion of the configuration model. Settings should transition to YAML once their meaning and validation rules are well-defined.

To enhance safety, Python scripts validate the YAML request, ensuring readability without compromising deployment security. Validating a configuration file involves checking for valid YAML syntax, preventing overlapping subnet ranges, ensuring correct AWS account assignments, and detecting any semantic errors. If validation is successful, Python translates the inputs into a format consumable by Terraform.

This division of responsibilities ensures that configuration files clearly outline the desired setup, while Python scripts validate and translate these configurations for Terraform to manage. Actions control the execution of these translations.

Validation processes are crucial but have limitations. Offline validation, although necessary, cannot guarantee the availability of S3 bucket names, confirm that the deployment identity possesses all required permissions, or validate AWS quota limits for additional resources. These checks necessitate live planning or service calls.

Separate Terraform states are maintained for each responsibility, with each root owning a unique state file. For example, 'databricks/dev/demo/01-infra.tfstate' handles AWS networking, IAM, storage, encryption, and monitoring responsibilities. 'databricks/dev/demo/02-account.tfstate' manages Databricks workspace creation, account groups, assignments, and budgeting.

'databricks/dev/demo/02-workspace-access.tfstate' controls workspace feature entitlements, and 'databricks/dev/demo/04-workspace.tfstate' oversees workspace configuration, policies, warehouses, and jobs. Each state file is associated with the specific deployment identifiers. The runner checks for keys outside the designated namespace and rejects any unauthorized modifications.

This state segregation ensures clear accountability and prevents unintended changes to resources outside the intended scope.