Generative Modeling: Learning the Distribution Behind the Data

Generative Modeling begins with a fundamental shift in perspective. Instead of memorizing individual observations, the goal is to learn the underlying probabilistic structure that produced those observations. By representing this structure through a learned distribution, the model can evaluate data and generate new samples that follow the same patterns. This mental model is centered around learning the distribution behind the data, and then using that distribution for both estimation and generation tasks.

To illustrate, imagine a dataset containing numerous dog images. Simply storing these images is insufficient for generating new dog images that do not exist within the training set. Instead, the model must capture the shared structure found across the observed examples and represent this structure as a probability distribution. The training data can be conceptualized as samples drawn from an unknown Data Distribution, denoted as p_data(x).

Although the exact form of p_data(x) is not known beforehand, only samples are observed from this distribution.

Therefore, Generative Modeling seeks to learn a Model Distribution, p_model(x), from these observations. New samples can subsequently be drawn from this learned distribution: generated samples ∼ p_model(x). Here, p_data(x) signifies the probability distribution governing real data, whereas p_model(x) signifies the distribution learned by the model.

The ultimate objective is for the Model Distribution to faithfully represent the Data Distribution sufficiently to enable both probability evaluation and the generation of new samples.

Both Density Estimation and Sample Generation share the same foundational framework. Density Estimation focuses on learning a distribution capable of evaluating probabilities for data. If the model accurately captures the structure of real data, samples resembling the real data will have higher probability, while samples that deviate from this structure will have lower probability.

Crucially, the model is not merely remembering specific training examples; it is representing the broader probabilistic structure common to all the data.

Sample Generation, on the other hand, utilizes the learned Model Distribution to create new samples. These generated samples need not match any instances in the training set; what matters is that they adhere to the structure represented by p_model(x). The overall process proceeds as follows: observed training data → learn the Model Distribution → density estimation → sample generation.

A common misconception is that random sampling suffices for generating meaningful data. However, in high-dimensional spaces like images, text, and audio, not every point corresponds to a realistic data sample. For instance, if pixel values were chosen uniformly at random, the outcome would unlikely resemble an actual image. The same applies to text; random character selection rarely yields coherent language.

In high-dimensional spaces, the number of possible configurations is vast, but realistic data occupies a minuscule portion of that space. Generation, therefore, involves not merely selecting values from a large space. The model must represent the restricted structure where real data actually exists. The true challenge lies in capturing the complex structure of real observations within this space.

Manifold Learning provides a relevant context here. The naive approach of generating data as simply choosing values from a large space is inadequate. A better approach is to focus on learning where realistic data is structured and then sampling according to that learned structure. This reveals that constructing complex high-dimensional distributions typically requires sufficient model capacity.

When simple models fall short in capturing this complexity, Deep Generative Models emerge as a powerful solution for representing more intricate data distributions.