The Mindset Shift: Data Cleaning in Data Science vs. Data Engineering

I initially intended to write about another tool I learned while working on data engineering, but I discovered something more fundamental. While exploring PySpark and creating a modular ETL pipeline, I began questioning why I was cleaning data differently than in my data science projects. I had spent a lot of time cleaning data for analysis and machine learning.

However, building production-oriented pipelines made me realize that the problems we were solving were actually quite different, leading to a significant mindset shift.

1. Core Objective: Analysis vs. Reliability

In data science, cleaning is typically driven by the needs of the analysis or model. Depending on the question and assumptions, you might investigate outliers, handle missing values, remove duplicates, transform distributions, or engineer new features. The primary goal is to make the data useful and appropriate for the analysis.

Data engineering has distinct constraints. When building an automatic pipeline feeding downstream systems, you must consider schema consistency, data contracts, failure handling, scalability, and reproducibility. The goal shifts from finding the "correct" way to clean data to ensuring reliable data processing. Instead of choosing universally correct cleaning methods, the focus is on explicit, repeatable, observable, and appropriate rules for the consuming systems.

2. Handling Nulls: Context Matters

Missing data is a problem in both data science and engineering. The approach depends on the reason for missing data and downstream consequences. In data science, you might impute missing age values using the median. In a data pipeline, the response depends on the field's role. For example, if customer_id is missing, it may need to be rejected or quarantined if required for customer identification.

However, if email is optional, it's better to let the record pass through as NULL. Not every imperfect value is bad data; understanding which fields are required, optional, and what the downstream system expects is crucial.

3. From Manual Inspection to Automated Data Quality

The difference becomes most evident when comparing manual inspection in a notebook to automated data quality checks in a pipeline. During data exploration, you might notice strange data, like negative values in a column. After investigation and transformation updates, you run the notebook again. This approach works well for exploration.

However, in a pipeline running nightly, you won't be present to notice such issues. Data quality checks should be part of the pipeline. Define rules, such as requiring non-null customer_id, valid transaction timestamps, non-negative revenue, and unique transaction IDs. Not every violation requires the same response. Critical errors might lead to record quarantine, while missing emails can be monitored without rejecting the record.

Unexpected but recoverable schema changes might trigger an alert instead of shutting down the entire pipeline.

4. Cleaning vs. Protecting the Data Pipeline

The most significant distinction lies in the purpose of cleaning. In data science, it's preparing a dataset for a specific analysis. In data engineering, transformations are part of creating a reliable data product. This involves questioning assumptions, considering possible violations, and deciding on record treatment—whether to transform, reject, quarantine, or allow through.

The pipeline must be designed to work reliably, even if the data is imperfect. The code may still contain familiar operations like filtering nulls, casting types, removing duplicates, and transforming columns. However, the context has changed. The focus is on building a system that keeps processing data reliably, regardless of imperfections.

The tooling question that arose from this shift is whether using Pandas for data engineering pipelines is feasible. While you may know how to clean and transform data with Pandas, the mindset shift highlights the need for a more robust and dependable approach when building production-oriented pipelines.