Deep Learning and Transformers

Artificial intelligence frequently introduces neural networks as "imitating the human brain." While this analogy holds value, those with backgrounds in physics, mathematics, engineering, or scientific computing can appreciate a different perspective. To modern AI practitioners, a neural network represents a parameterized mathematical transformation.

During training, the network navigates an optimization problem within an extremely high-dimensional space. The attention mechanism inside a Transformer can be viewed as a learned, input-dependent interaction matrix. With this viewpoint, intricate AI behaviors become more comprehensible.

Let's dissect this concept from the ground up. Start with an artificial neuron, the fundamental computational element of a neural network. Assume multiple inputs: x₁, x₂, ..., xₙ. Each input is tied to a weight: w₁, w₂, ..., wₙ. The neuron computes a weighted sum: z = Σᵢ wᵢxᵢ + b, where b is the bias. This result then undergoes an activation function, producing: a = f(z).

In vector notation, the same idea is expressed as: z = w · x + b. Although named after biological neurons, artificial neurons are fundamentally mathematical operations. Their true power emerges when these operations are connected in layers.

Why do we require activation functions? If we construct multiple neural-network layers using linear transformations, the first layer might be: h₁ = W₁x. The subsequent layer could be: h₂ = W₂h₁. By substituting the first expression into the second, we obtain: h₂ = W₂W₁x. However, W₂W₁ remains simply another matrix. Stacking numerous purely linear layers merely results in one larger linear transformation.

The incorporation of nonlinearity changes this. Suppose the first layer transforms to: h₁ = f(W₁x + b₁) and the next to: h₂ = f(W₂h₁ + b₂). The activation function f prevents the entire network from reducing to a single linear transformation. The ReLU activation function, ReLU(x) = max(0, x), is a common choice. If the input is positive, ReLU retains it; if negative, it returns zero.

This basic nonlinearity enables networks to represent much more intricate relationships, forming a cornerstone of deep learning.

Deep neural networks consist of numerous transformation stages. Information traverses a path like this: x ↓ f(W₁x + b₁) ↓ f(W₂h₁ + b₂) ↓ ... ↓ y. Each layer refines the representation from the preceding layer. An image-processing system might initially detect simple structures such as edges and color variations in early layers. Subsequent layers can amalgamate these components into more complex patterns, which in turn can be combined to yield increasingly valuable internal representations.

In essence, a deep neural network embodies a highly dimensional parameterized function. Its architecture dictates the function's structure, while training determines the numerical values of its parameters.

Training a network involves addressing an optimization problem. Suppose the network's prediction is ŷ, while the desired output is y. We define a loss function, L(ŷ, y), to quantify the discrepancy between prediction and desired result. Training then seeks the parameter values that minimize this loss: θ* = arg minθ L(θ). Conceptually, we aspire to find a parameter configuration that minimizes the loss.

Visualizing this loss function over billions of parameters is challenging, but conceptualizing it as a landscape with regions of varying loss is useful. From a physics perspective, neural-network training resembles optimization in a vast-dimensional landscape. Rather than explicitly encoding every rule, we explore parameter values that enable the model to capture useful data patterns.

How do we navigate this immense parameter space? We employ gradient descent. The gradient of the loss, ∇θ L, reveals how the loss varies with small parameter adjustments. The gradient descent update rule is: θₜ₊₁ = θₜ − η∇θL, where θₜ denotes the current parameters, ∇θL represents the gradient of the loss, η is the learning rate, and θₜ₊₁ signifies the updated parameters.

The learning rate controls the magnitude of each step, balancing the need to avoid overshooting beneficial regions and the potential for excessively slow training. Modern training algorithms have evolved beyond basic gradient descent, but this fundamental concept endures.