Modeling Data Distributions: The Core Idea Behind Generative Models

A generative model aims to capture the probability structure that generated a dataset, rather than simply memorizing the data itself. This core idea forms the foundation of distribution modeling, as we observe finite samples but lack knowledge of the exact distribution behind them.

To address this challenge, we introduce a parameterized Model Distribution, denoted as p model (x; θ), which aims to approximate the unknown Data Distribution, p data (x). The goal is to learn the parameters θ of the Model Distribution from the observed samples.

Training a generative model can be particularly difficult when dealing with high-dimensional data. A well-designed probability model must strike a balance between sufficient flexibility to represent complex structures and enough tractability to be evaluated and trained effectively. This tradeoff between Tractability and Flexibility often motivates the development of more expressive Deep Generative Models.

Let's consider a training dataset containing N observations, denoted as x1, x2, ..., xN, which are independently drawn from an unknown Data Distribution. We do not know the analytical form of this underlying distribution; we only have access to the observed samples. For instance, in the case of image datasets, we directly observe the image files, but we are not privy to the probability structure that determines the prevalence or rarity of certain image patterns.

The relationship between the unknown Data Distribution and the observed samples can be summarized as: unknown Data Distribution ⊢ observed samples. Collecting a large number of samples allows us to uncover recurring statistical patterns and shared structures, which can then be used to estimate the properties of the underlying distribution.

However, since we do not have direct access to the true Data Distribution, we cannot rely on it to evaluate probabilities or generate new samples. Instead, we rely on a separate probability model, the Model Distribution, denoted as p model (x; θ), which we learn from the observed data. The Model Distribution serves as our proxy for the unknown Data Distribution.