Beyond the Basics: What I learned from a Deep-Dive EDA on the Ames Housing Dataset

In a recent deep-dive Exploratory Data Analysis (EDA) on the Ames, Iowa Housing Dataset, a ML enthusiast uncovered several key insights and counter-intuitive findings. The Ames dataset, known as the "final boss" version of the Boston Housing dataset, consists of 2,930 residential properties and over 80 explanatory variables.

One of the primary takeaways was the importance of smarter Feature Engineering. The analyst initially worked with square footage data split across multiple columns, but created composite features like Total_Usable_SF, which combined Above-Grade Living Area and Total Basement square footage. This provided a more realistic representation of a home's total usable space.

Additionally, they introduced Price_Per_SqFt, a normalized metric created by dividing the Sale Price by Total_Usable_SF, which facilitated easier comparisons between different neighborhoods and removed biases related to house size. To address the right-skewed distribution of Sale_Price, caused by a few multi-million dollar properties, the analyst applied a logarithmic transformation using np.log1p, which normalized the distribution and is crucial for linear modeling assumptions.

The analyst also discussed the significance of missing data in the housing context. Unlike other datasets, missing values in housing often indicate the absence of a feature rather than simply missing data. For instance, missing values for Pools, Fences, and Garages were converted into binary presence/absence flags. Houses without a Basement or Garage faced a significant median price penalty of 65% to 85% or more.

This was particularly notable in Iowa, where such features are almost exclusively found in low-end starter homes. The analyst found it counter-intuitive that features often associated with upgrades, like Fences or Alley access, actually correlated with a lower median sale price. This was likely a proxy for age and location, as these features are common in older, denser urban tracts of Ames, while expensive, newly built developments do not have them.

Lastly, the analyst tackled the challenge of extreme outliers, specifically Pools. Despite their significant price premium, Pools were present in only 13 out of 2,930 homes (0.44%), making them an ultra-luxury edge case. The question arose: how to handle features that exist in less than 1% of the dataset? Should they be kept as binary flags due to their predictive power on high-end outliers, or dropped completely due to sparsity before feeding them into a model like XGBoost? The analyst invited feedback on this dilemma in the comments section.