Learning neural networks through mental models

When I first grasped neural networks, I comprehended them in a mechanical manner. Data entered the network, passed through several layers, and ultimately generated a prediction. Loss was calculated, gradients computed, and weights updated. While I could follow the sequence, the individual components still seemed somewhat arbitrary.

The reason behind their structure, the power of stacking layers, and the significance of activation functions remained unclear. Although I understood much of what the network did step by step, I lacked a sense of comprehending the network holistically. Then, I watched Josh Starmer’s StatQuest video on the fundamental ideas underlying neural networks.

He portrayed neural networks as systems that shape functions. Instead of focusing on data traversing layers, I could step back and envision the collective construction of those transformations. This shift in perspective immediately helped. Subsequently, I was eager to explore additional ways to conceptualize neural networks. By the time I had finished, I realized that various mental models made different questions easier to address.

Some mental models allowed me to zoom out and consolidate the network into a single idea. Others broke it down into smaller transformations or computations. Some connected neural networks to concepts I already understood; others compelled me to reinterpret preconceived notions. Collectively, these mental models made neural networks feel far less enigmatic.

The breakthrough came from learning to switch between these mental models. Viewing neural networks as stacks of transformations was my initial mechanical understanding. Each layer progressively transformed the data before passing it on: where x represented the input, h's were intermediate hidden representations, and y was the final output.

This perspective made the forward pass easy to follow: the input underwent incremental transformations until the network produced an output. This viewpoint elucidated why neural networks have depth. Each layer could transform the output of the preceding layer, enabling the network to construct increasingly intricate computations.

Simultaneously, this perspective kept me focused on individual steps, preventing a clear picture of the collective impact of these transformations. I needed a higher-level view. Function approximation provided that broader perspective. I was already familiar with models like linear and logistic regression as functions. Given specific inputs, these models learned a mapping that produced an output.

A neural network could be understood similarly: it learned a complex function that maps inputs to outputs, akin to other models I already comprehended. The difference lay in the flexibility and expressiveness of the functions neural networks could learn. But how did these intricate functions emerge from simple neurons? While I abstractly understood that stacking layers with nonlinear activation functions enabled neural networks to model increasingly complex relationships, I couldn’t visualize this process.

In Josh Starmer’s explanation, each neuron functioned with a given activation function. The weights and biases sliced, flipped, and stretched that same activation function into new shapes. Consequently, each neuron applied a minor nonlinear transformation to its input. These minor transformations and shapes were intertwined across layers to produce even more complex shapes.

A sophisticated decision boundary emerged from numerous localized transformations. Instead of learning one massive function directly, neural networks built complex behavior by combining numerous smaller nonlinear transformations. Rather than learning one complex function directly, neural networks constructed complexity by combining many smaller nonlinear transformations.

This realization was profound. It didn't require any individual neuron to represent a complicated function. Each neuron could contribute something simple. The complexity arose from composing and combining these simple transformations. The concept of function approximation had helped me zoom out and perceive the network as one flexible function.

However, function sculpting allowed me to zoom in just enough to grasp how this flexibility could be constructed. Another perspective I found illuminating was that of neural networks as representation learners. Unlike many models I had previously used in econometrics and causal inference, neural networks have the ability to learn how to represent the input data itself.

In my previous experience, deciding how to represent a problem typically occurred before fitting the model. I carefully considered which variables to include, what they measured, and how to transform them. The model then learned from the representation I had provided. In contrast, with neural networks, determining how to represent the problem doesn't happen entirely beforehand.

The network also learns useful ways of representing the data during training. The hidden states in the stack of transformations aren't merely intermediate values; they are intermediate representations shaped during training to become beneficial for the task. This is why neural networks are often described as systems for representation learning.

This concept applies even to relatively simple neural networks. For instance, a churn model might learn hidden behavioral patterns that distinguish customers with varying risks of leaving. In a larger language model, the learned representations could encode intricate semantic relationships.