Discriminative and generative models compared

Broadly speaking, machine learning models can be subdivided into discriminative models and generative models. Discriminative models learn a map from some input to some output. In discriminative models, learning the process that generates the input is not relevant; it will just learn a map from the to the expected output.

Generative models, on the other hand, in addition to learning a map from some input to some output, also learn the process that generates the input and the output. 

In this context, we say that discriminative models estimate : the conditional probability distribution of  conditioned on . Note that, in this case, the input x is fixed, known a priori, and the discriminative model estimates the probability of , _, but does not have any information about the marginal distribution of  nor  independent of the other variables,  and  respectively. Discriminative models can be used to learn a map, _,that produces outputs that approximates the distribution . 

Generative models, on the other hand, estimate : the joint probability distribution of x and y. The joint probability distribution is symmetric and can be written as . Generative models can be used to learn a map, _,that  approximates the distribution . 

By estimating  and knowing that the joint probability is symmetrical, we can use Bayes' theorem to obtain  and _,as we briefly describe below:

By moving around the terms and on the second and third equation we derive  Bayes' theorem: