What is the difference between a generative and a discriminative algorithm?

Question

What is the difference between a generative and a discriminative algorithm?

Answer 1

An addition informative point that goes well with the answer by StompChicken above.

The fundamental difference between discriminative models and generative models is:

Discriminative models learn the (hard or soft) boundary between classes

Generative models model the distribution of individual classes

Edit:

A Generative model is the one that can generate data. It models both the features and the class (i.e. the complete data).

If we model P(x,y): I can use this probability distribution to generate data points - and hence all algorithms modeling P(x,y) are generative.

Eg. of generative models

• Naive Bayes models P(c) and P(d|c) - where c is the class and d is the feature vector.

  Also, P(c,d) = P(c) * P(d|c)

  Hence, Naive Bayes in some form models, P(c,d)

• Bayes Net

• Markov Nets

A discriminative model is the one that can only be used to discriminate/classify the data points. You only require to model P(y|x) in such cases, (i.e. probability of class given the feature vector).

Eg. of discriminative models:

• logistic regression

• Neural Networks

• Conditional random fields

In general, generative models need to model much more than the discriminative models and hence are sometimes not as effective. As a matter of fact, most (not sure if all) unsupervised learning algorithms like clustering etc can be called generative, since they model P(d) (and there are no classes:P)

PS: Part of the answer is taken from source