Stop bad data at the pull request with Great Expectations and GitHub Actions

In this post, a pull request will fail in GitHub if it breaks the data quality before anyone can merge it. Traditionally, data quality checks are performed after the data has landed, but issues can arise from code changes that were approved just an hour earlier. To address this, three components are implemented: a data contract in YAML, a Python script that transforms it into Great Expectations (GX) checks, and a GitHub Actions workflow that runs the script on every pull request.

The necessary files are provided, including Python 3.12, a GitHub repository, and the requirements.txt file with the following packages: great_expectations==1.23.2, pandas==2.3.3, and pyyaml==6.0.3. The GX 1.x API is used, which is different from the older 0.x version.

Step 1 involves building the pipeline where raw orders with messy status values are cleaned up. The pipeline/build_orders.py file contains a function called build_orders that takes raw order events as input and returns a clean `orders` table. The function removes any rows with a cancelled status, converts the status column to lowercase, and calculates the total amount for each order by multiplying the quantity by the unit price.

Step 2 focuses on creating a data contract using a YAML file named contracts/orders.yml. This contract defines the structure of the `orders` table and sets rules for each column. The contract is version-controlled and reviewed by the team before any changes are approved. The columns specified in the contract include order_id, customer_id, status, total_amount, currency, and created_at, each with specific requirements and constraints such as required fields, allowed values, and minimum and maximum values.

Step 3 covers the process of converting the YAML contract into GX checks. The first part of the validate_contract.py script transforms each rule from the contract into a GX Expectation. It imports necessary libraries, including great_expectations, pandas, and yaml, and defines a function called expectations_from_contract. This function takes the contract dictionary as input and returns a list of GX Expectations.

For each column in the contract, it checks if the column is required, unique, has allowed values, or follows a specific pattern. Based on these conditions, it appends the corresponding GX Expectation to the list. This step ensures that the data adheres to the defined rules before merging into the main branch.