Which machine learning classifier to choose, in general? [closed]

Answer 1

Sam Roweis used to say that you should try naive Bayes, logistic regression, k-nearest neighbour and Fisher's linear discriminant before anything else.

Answer 2

You should always keep into account the inference vs prediction trade-off.

If you want to understand the complex relationship that is occurring in your data then you should go with a rich inference algorithm (e.g. linear regression or lasso). On the other hand, if you are only interested in the result you can go with high dimensional and more complex (but less interpretable) algorithms, like neural networks.

Answer 3

The book "OpenCV" has a great two pages on this on pages 462-463. Searching the Amazon preview for the word "discriminative" (probably google books also) will let you see the pages in question. These two pages are the greatest gem I have found in this book.

In short:

• Boosting - often effective when a large amount of training data is available.

• Random trees - often very effective and can also perform regression.

• K-nearest neighbors - simplest thing you can do, often effective but slow and requires lots of memory.

• Neural networks - Slow to train but very fast to run, still optimal performer for letter recognition.

• SVM - Among the best with limited data, but losing against boosting or random trees only when large data sets are available.

Answer 4

Selection of Algorithm is depending upon the scenario and the type and size of data set. There are many other factors.

This is a brief cheat sheet for basic machine learning.

Answer 5

Things you might consider in choosing which algorithm to use would include:

1. Do you need to train incrementally (as opposed to batched)?

   If you need to update your classifier with new data frequently (or you have tons of data), you'll probably want to use Bayesian. Neural nets and SVM need to work on the training data in one go.

2. Is your data composed of categorical only, or numeric only, or both?

   I think Bayesian works best with categorical/binomial data. Decision trees can't predict numerical values.

3. Does you or your audience need to understand how the classifier works?

   Use Bayesian or decision trees, since these can be easily explained to most people. Neural networks and SVM are "black boxes" in the sense that you can't really see how they are classifying data.

4. How much classification speed do you need?

   SVM's are fast when it comes to classifying since they only need to determine which side of the "line" your data is on. Decision trees can be slow especially when they're complex (e.g. lots of branches).

5. Complexity.

   Neural nets and SVMs can handle complex non-linear classification.

Answer 6

My take on it is that you always run the basic classifiers first to get some sense of your data. More often than not (in my experience at least) they've been good enough.

So, if you have supervised data, train a Naive Bayes classifier. If you have unsupervised data, you can try k-means clustering.

Another resource is one of the lecture videos of the series of videos Stanford Machine Learning, which I watched a while back. In video 4 or 5, I think, the lecturer discusses some generally accepted conventions when training classifiers, advantages/tradeoffs, etc.