Discriminative vs Generative Models: What Are You Actually Learning?

Discriminative and generative models are two distinct approaches to learning mappings from inputs to outputs in supervised learning. While they share the same prediction target, they model different probability relationships and arrive at predictions through different calculation flows.

Discriminative models directly learn the conditional probability of the output given the input, p(y|x). In classification tasks, the prediction is typically the class with the highest probability, f(x) = argmax_y p(y|x). The model focuses on finding the decision boundary that separates different outputs based on the input space. Support Vector Machines are a classic example of discriminative models that learn this boundary.

On the other hand, generative models model the probability of the input given each possible output, p(x|y). They then combine this conditional probability with the prior probability of each output, p(y), to make predictions. The prediction can be expressed as f(x) = argmax_y p(x|y)p(y). By evaluating p(x|y) for each candidate output and combining it with p(y), the model selects the output with the highest product.

This approach connects back to the conditional probability used in discriminative models through Bayes' rule: p(y|x) = p(x)p(x|y)/p(x).

The key difference lies in the structure the models learn. Discriminative models concentrate on directly learning the relationship between inputs and outputs, while generative models capture the distribution of inputs for each output. This distinction affects how the models approach the prediction problem and can lead to different strengths and weaknesses depending on the specific task and data.