Generative Modeling: From Data Distributions to Deep Generative Models

Generative modeling tackles the challenge of representing and learning complex high-dimensional probability distributions from data. The field can be organized around three key problems: representation, learning, and inference.

Representation involves determining how to model the intricate joint distribution of high-dimensional data. Learning focuses on making the model distribution converge to the true data distribution. Inference deals with reasoning backward from observations to uncover hidden variables or the processes that generated them.

While generative models may initially appear as disparate architectures focused on image generation, a more useful perspective is to start at a lower level: understanding what probability structure could have produced the data and how to represent, learn, and infer that structure without making computation intractable. This approach connects various models, including autoregressive models, VAEs, flow-based models, GANs, and diffusion models.

At its core, generative modeling seeks to address three intertwined problems: how to represent a complex high-dimensional distribution, how to make the model distribution approximate the data distribution, and how to reason backward from observations to hidden variables or the generating process. By connecting these components, the major families of deep generative models become more comprehensible.

Discriminative models typically begin with a prediction problem, aiming to predict the most likely output given an input. They directly focus on the conditional relationship required for prediction. In contrast, generative models take a broader approach. Instead of merely learning the path from input to output, they model the underlying probability structure from which the data originates.

For class-conditional modeling, the model learns both the conditional distribution p(x|y) and the prior distribution p(y), enabling the recovery of the posterior distribution p(y|x) using Bayes' rule.

In unsupervised generative modeling, the target becomes the data distribution itself. The training samples are assumed to come from an unknown distribution: x1, x2, ..., xN ~ i.i.d. p_data(x). The model constructs a parameterized distribution that aims to approximate this unknown distribution. This perspective bridges tasks that might otherwise seem separate, such as learning from training data to generate new examples.