Why scikit-learn neighbors is slower with n_jobs > 1 and forkserver

问题

I'm using scikit-learn for doing Metaheuristics exercises and I have a doubt: I need to use knn, so I have a KNearestNeighbors object with n_jobs=-1. As the docs said, I have to set the multiprocessing mode to forkserver. But the knn is soooo slower with n_jobs=-1 than with n_jobs=1.

This is some piece of code

### Some initialization here ###
skf = StratifiedKFold(target, n_folds=2, shuffle=True)

for train_index, test_index in skf:
       data_train, data_test = data[train_index], data[test_index]
       target_train, target_test = target[train_index], target[test_index]

       start = time()
       selected_features, score = SFS(data_train, data_test, target_train, target_test, knn)
       end = time()

       logger.info("SFS - Time elapsed: " + str(end-start) + ". Score: " + str(score) + ". Selected features: " + str(sum(selected_features)))
if __name__ == "__main__":
    import multiprocessing as mp; mp.set_start_method('forkserver', force = True)
    main()

This is the SFS function

def SFS(data_train, data_test, target_train, target_test, classifier):
    rowsize = len(data_train[0])
    selected_features = np.zeros(rowsize, dtype=np.bool)
    best_score = 0
    best_feature = 0

    while best_feature is not None:
        end = True
        best_feature = None

        for idx in range(rowsize):
            if selected_features[idx]:
                continue

            selected_features[idx] = True
            classifier.fit(data_train[:,selected_features], target_train)
            score = classifier.score(data_test[:,selected_features], target_test)
            selected_features[idx] = False

            if score > best_score:
                best_score = score
                best_feature = idx

        if best_feature is not None:
            selected_features[best_feature] = True

    return selected_features, best_score

I don't understand how can n_jobs > 1 be slower than n_jobs = 1. Can anyone explain me that? I've tried with 3 dataset.

回答1:

I found out many of people like you had same problem : n_jobs is not working in KNearestNeighbors of sklearn. And they also complained that just 1 CPU core was loaded.

In my experiment, fitting process uses just single core whether n_jobs>1 or not. So whether you set n_jobs as large number, if your train data sample is large, the time for training will be huge and not reduced.

And the reason n_jobs>1 is even more slow than n_jobs=1 is because of the cost to distribute resources for multiprocessing.

来源：https://stackoverflow.com/questions/36250555/why-scikit-learn-neighbors-is-slower-with-n-jobs-1-and-forkserver