问题
I tried to do feature selection with RFECV but it is giving out different results each time, does cross-validation divide the sample X into random chunks or into sequential deterministic chunks?
Also, why is the score different for grid_scores_ and score(X,y)? why are the scores sometimes negative?
回答1:
Does cross-validation divide the sample X into random chunks or into sequential deterministic chunks?
CV divides the data into deterministic chunks by default. You can change this behaviour by setting the shuffle parameter to True.
However, RFECV uses sklearn.model_selection.StratifiedKFold if the y is binary or multiclass.
This means that it will split the data such that each fold has the same (or nearly the same ratio of classes). In order to do this, the exact data in each fold can change slightly in different iterations of CV. However, this should not cause major changes in the data.
If you are passing a CV iterator using the cv parameter, then you can fix the splits by specifying a random state. The random state is linked to random decisions made by the algorithm. Using the same random state each time will ensure the same behaviour.
Also, why is the score different for grid_scores_ and score(X,y)?
grid_scores_ is an array of cross-validation scores. grid_scores_[i] is the cross-validation score for the i-th iteration. This means that the first score is the score for all features, the second is the score when one set of features is removed and so on. The number of features removed in each is equal to the value of the step parameter. This is = 1 by default.
score(X, y) selects the optimal number of features and returns the score for those features.
why are the scores sometimes negative?
This depends on the estimator and scorer you are using. If you have set no scorer RFECV will use the default score function for the estimator. Generally, this is accuracy, but in your particular case, might be something that returns a negative value.
来源：https://stackoverflow.com/questions/47374254/why-is-sklearn-feature-selection-rfecv-giving-different-results-for-each-run