Kernel Mean Embeddings: Turning Distributions into Points You Can Compute With

Kernel mean embeddings are a technique that allows us to represent distributions using a single point in a high-dimensional function space. This is a significant improvement over traditional methods that either require complex integrals, density estimates, or parametric assumptions to compare or work with distributions. By embedding the entire distribution as a single point, we can perform operations on distributions as if they were simple points, making things much easier to handle computationally.

The key idea behind kernel mean embeddings is to use a kernel function to measure the similarity between two points. The kernel function k(x, y) is related to an inner product in a higher-dimensional feature space, which is represented by the reproducing kernel Hilbert space (RKHS). Instead of explicitly computing the feature map ϕ(x), the kernel function does the work for us.

Now, suppose we have a distribution P instead of a single point. The natural analogue of the feature map ϕ(x) in this case is the kernel mean embedding of P. This is defined as the expectation of the feature map applied to random variables drawn from the distribution: μP = E[X~P][ϕ(X)]. In other words, μP is the "center of mass" of the distribution after it has been mapped into the feature space.

If the kernel is rich enough, different distributions will map to different points in the space, providing a faithful representation of the distribution rather than a lossy summary like a mean and variance.

One of the most significant advantages of kernel mean embeddings is that many operations on distributions can now be performed as operations on these points. For example, the expectation of any function f in the RKHS can be computed using an inner product with μP: E[X~P][f(X)] = ⟨f, μP⟩H. This means that evaluating expectations, which are typically integrals over the distribution P, becomes a simple inner product with the kernel mean embedding. This identity forms the basis for many applications of kernel mean embeddings.

In practice, we rarely know the distribution P; we only have samples from it, such as x1, x2, ..., xn. However, the kernel mean embedding approach allows us to work with these samples effectively, as the operations on distributions can still be carried out using the computed kernel mean embeddings.