Hierarchical Clustering Fails Beautifully

The most striking outcome during an initial exploration of Week-1 data did not emerge from poor predictions. Instead, it came in the form of a stunning tree structure produced by hierarchical clustering. Such dendrograms elegantly depict how various entities—accounts, tickets, or incidents—nest within broader categories. They appear to reveal hidden truths, captivating stakeholders who often screenshot them for presentations. However, beneath their convincing appearance lies the potential for significant inaccuracies.

Hierarchical clustering operates on a simple principle: start with each data point as its own cluster and then iteratively merge the closest pairs until a desired structure is achieved. This greedy, irreversible process resembles the growth of a decision tree. An alternative approach, divisive clustering, works in reverse by beginning with all data points together and splitting them apart.

Two key aspects of hierarchical clustering are often overlooked. First, the decision to cut the dendrogram at a particular level is made after examining the resulting structure, rather than being determined upfront. This flexibility allows the number of clusters to be tailored to specific needs, reflecting an organizational hierarchy that is already familiar to many in a delivery environment.

Second, the choice of linkage criterion—the method used to determine the distance between clusters—is a selectable worldview, influencing the shape and interpretation of the resulting tree.

Different linkage methods—Ward's minimum variance, single linkage, and complete linkage—each embody distinct assumptions about cluster formation. Ward's method, for instance, favors compact clusters by minimizing the increase in within-cluster variance at each merge. Meanwhile, single linkage's emphasis on chaining and complete linkage's preference for tighter groups provide alternative perspectives on grouping data.

One interesting connection emerges when considering single linkage as a minimum spanning tree—a graph algorithm that connects points with the cheapest edges before eliminating the longest ones. This realization highlights how hierarchical clustering can sometimes rely on familiar graph concepts to explain its behavior.

Another unexpected insight comes from understanding what the tree is actually optimizing. Rather than seeking a single global objective, hierarchical clustering makes greedy decisions based on the chosen linkage criterion. For example, Ward's method minimizes the increase in within-cluster sum of squares at each merge, mirroring the objective used in K-Means clustering. Consequently, neither approach guarantees the optimal solution.

A critical distinction lies in the fact that Ward's method is limited to Euclidean distance. Non-Euclidean distances, such as cosine similarity, are not compatible with Ward's approach, which restricts its use to Euclidean/L2 distance metrics.

The interpretation of dendrograms is also subject to bias. The left-to-right positioning of leaves does not inherently convey similarity information. Instead, the vertical axis reveals how far apart groups were when they merged. However, due to common reading habits, people often mistakenly assume that neighboring leaves are similar, when in reality, they may only merge at higher levels of the tree, indicating greater dissimilarity.

To assess the reliability of a dendrogram, two useful checks can be employed. First, the cophenetic correlation compares the original pairwise distances with those represented by the dendrogram, providing a measure of how well the tree preserves the original geometry. Low correlation suggests significant distortion. Second, cluster stability assesses whether the clustering results remain consistent when the data is resampled and the clustering process is repeated.

If the membership of data points changes substantially across resamples, the dendrogram is likely to be unreliable.

Standardizing the data before fitting hierarchical clustering is also crucial, as distances are sensitive to scale. Finally, when multiple merges have equal criteria, implementation details and input ordering can lead to variations in the resulting tree structure, emphasizing the importance of consistent methods when comparing different dendrograms.