Which machine learning classifier to choose, in general? [closed]

Answer 1

First of all, you need to identify your problem. It depends upon what kind of data you have and what your desired task is.

If you are Predicting Category :

• You have Labeled Data
  • You need to follow Classification Approach and its algorithms
• You don't have Labeled Data
  • You need to go for Clustering Approach

If you are Predicting Quantity :

• You need to go for Regression Approach

Otherwise

• You can go for Dimensionality Reduction Approach

There are different algorithms within each approach mentioned above. The choice of a particular algorithm depends upon the size of the dataset.

Source: http://scikit-learn.org/stable/tutorial/machine_learning_map/

Answer 2

As Prof Andrew Ng often states: always begin by implementing a rough, dirty algorithm, and then iteratively refine it.

For classification, Naive Bayes is a good starter, as it has good performances, is highly scalable and can adapt to almost any kind of classification task. Also 1NN (K-Nearest Neighbours with only 1 neighbour) is a no-hassle best fit algorithm (because the data will be the model, and thus you don't have to care about the dimensionality fit of your decision boundary), the only issue is the computation cost (quadratic because you need to compute the distance matrix, so it may not be a good fit for high dimensional data).

Another good starter algorithm is the Random Forests (composed of decision trees), this is highly scalable to any number of dimensions and has generally quite acceptable performances. Then finally, there are genetic algorithms, which scale admirably well to any dimension and any data with minimal knowledge of the data itself, with the most minimal and simplest implementation being the microbial genetic algorithm (only one line of C code! by Inman Harvey in 1996), and one of the most complex being CMA-ES and MOGA/e-MOEA.

And remember that, often, you can't really know what will work best on your data before you try the algorithms for real.

As a side-note, if you want a theoretical framework to test your hypothesis and algorithms theoretical performances for a given problem, you can use the PAC (Probably approximately correct) learning framework (beware: it's very abstract and complex!), but to summary, the gist of PAC learning says that you should use the less complex, but complex enough (complexity being the maximum dimensionality that the algo can fit) algorithm that can fit your data. In other words, use the Occam's razor.

Answer 3

First of all, it depends on which type of problem you're dealing with whether it's classification or regression. Then choose your model wisely. It depends on a particular model one specific model outperform other models. Suppose you are working on wine_dataset from sklearn library and first you tried to train the data with kernel svm with linear and you get some sort of accuracy and then you think that it is not satisfying so then you tried to train your data with DecisionTreeClassifier() and then you tried with RandomForestClassifier(). After that whichever accuracy will be better or you can say that fits your data, you can conclude that. There is so small syntactical difference you will find while keep on changing model for testing. So all best and understand the problem well.

Answer 4

Model selection using cross validation may be what you need.

Cross validation

What you do is simply to split your dataset into k non-overlapping subsets (folds), train a model using k-1 folds and predict its performance using the fold you left out. This you do for each possible combination of folds (first leave 1st fold out, then 2nd, ... , then kth, and train with the remaining folds). After finishing, you estimate the mean performance of all folds (maybe also the variance/standard deviation of the performance).

How to choose the parameter k depends on the time you have. Usual values for k are 3, 5, 10 or even N, where N is the size of your data (that's the same as leave-one-out cross validation). I prefer 5 or 10.

Model selection

Let's say you have 5 methods (ANN, SVM, KNN, etc) and 10 parameter combinations for each method (depending on the method). You simply have to run cross validation for each method and parameter combination (5 * 10 = 50) and select the best model, method and parameters. Then you re-train with the best method and parameters on all your data and you have your final model.

There are some more things to say. If, for example, you use a lot of methods and parameter combinations for each, it's very likely you will overfit. In cases like these, you have to use nested cross validation.

Nested cross validation

In nested cross validation, you perform cross validation on the model selection algorithm.

Again, you first split your data into k folds. After each step, you choose k-1 as your training data and the remaining one as your test data. Then you run model selection (the procedure I explained above) for each possible combination of those k folds. After finishing this, you will have k models, one for each combination of folds. After that, you test each model with the remaining test data and choose the best one. Again, after having the last model you train a new one with the same method and parameters on all the data you have. That's your final model.

Of course, there are many variations of these methods and other things I didn't mention. If you need more information about these look for some publications about these topics.