Nine Algorithms Later, I Stopped Learning Algorithms

The final article in the series about machine learning algorithms concluded that the focus had shifted from simply learning and memorizing the algorithms to understanding their underlying principles and assumptions. The author, previously an SRE and DevOps professional, found themselves questioning the fundamental aspects of machine learning, such as the algorithm's assumptions about the data, what it considers incorrect, and how it searches for a less incorrect answer.

The author identified three key questions to guide their analysis: what shape does the algorithm assume the world has, what does it consider wrong, and how does it go looking for a less incorrect answer? These questions correspond to the algorithm's hypothesis, loss function, and optimization method. However, the author quickly found themselves making mistakes in distinguishing these components across the various algorithms they studied.

As the series progressed, the author discovered five distinct approaches algorithms use to arrive at their results: K-Means searches, DBSCAN definitions, PCA solutions, hierarchical clustering builds, and Isolation Forest randomizations. These methods illustrate the diversity of strategies algorithms employ to solve problems.

The author also reflected on the role of human judgment in machine learning. While algorithms optimize certain aspects, the human-defined parameters and configurations often play a crucial role in determining the model's behavior. The author observed that the less an algorithm optimizes, the more the human setup becomes the driving force behind its outcomes.

Some properties of algorithms may initially appear problematic but are actually necessary for their proper functioning. For example, PCA components are defined only up to a sign, and hierarchical clustering dendrograms can have different left-to-right orders without changing the underlying hierarchy. Recognizing these nuances is essential for accurate interpretation and comparison of machine learning models.

In production systems, the configuration surrounding the algorithms is often as important as the algorithms themselves. The author emphasized that the properties that can look like bugs may, in fact, be essential features of the algorithms. Understanding these subtleties allows for a more nuanced approach to machine learning, where the focus shifts from rote memorization to a deeper comprehension of the algorithms' underlying principles and assumptions.