RFECV does not return same features for same data

Question

I have a dataframe X which is comprised of 60 features and ~ 450k outcomes. My response variable y is categorical (survival, no survival).

I would like to use RFECV to reduce the number of significant features for my estimator (right now, logistic regression) on Xtrain, which I would like to score of accuracy under an ROC Curve. "Features Selected" is a list of all features.

from sklearn.cross_validation import StratifiedKFold
from sklearn.feature_selection import RFECV
import sklearn.linear_model as lm

#  Create train and test datasets to evaluate each model
Xtrain, Xtest, ytrain, ytest = train_test_split(X,y,train_size = 0.70)
# Use RFECV to reduce features

#  Create a logistic regression estimator 
logreg = lm.LogisticRegression()

# Use RFECV to pick best features, using Stratified Kfold
rfecv =   RFECV(estimator=logreg, cv=StratifiedKFold(ytrain, 10), scoring='roc_auc')

# Fit the features to the response variable
X_new = rfecv.fit_transform(Xtrain[features_selected], ytrain)

I have a few questions:

a) X_new returns different features when run on separate occasions (one time it returned 5 features, another run it returned 9. One is not a subset of the other). Why would this be?

b) Does this imply an unstable solution? While using the same seed for StratifiedKFold should solve this problem, does this mean I need to reconsider the approach in totality?

c) IN general, how do I approach tuning? e.g., features are selected BEFORE tuning in my current implementation. Would tuning affect the significance of certain features? Or should I tune simultaneously?

a & b) I would bet that if you use LeaveOneOut CV, you will get a more stable result c) IMO try both and see how it affects CV score. As a general comment, there are other ways to perform feature selection that are worth looking into. For example, check out my answer at stackoverflow.com/questions/29230019/… — Ryan
@Ryan - thanks for the help. I'll check LOOCV. Regarding your answer to question 29230019, what would you suggest if Feature Importance is not accessible? — GPB

armaro armaro · Accepted Answer · 2018-05-28T11:31:05

In k-fold cross-validation, the original sample is randomly partitioned into k equal size sub-samples. Therefore, it's not surprising to get different results every time you execute the algorithm. Source

There is an approach, so-called Pearson's correlation coefficient. By using this method, you can calculate the a correlation coefficient between each two features, and aim for removing features with a high correlation. This method could be considered as a stable solution to such a problem. Source

RFECV does not return same features for same data

1 Answers