Recursive feature elemination with CV doesn't reduce feature count

Question

I have this protein dataset that I need to perform a RFE on. There are 100 examples with binary class labels (sick - 1, healthy - 0) and 9847 features for each example. To reduce the dimensionality I am performing a RFECV with a LogisticRegression estimator and 5 fold CV. This is the code:

model = LogisticRegression()
rfecv = RFECV(estimator=model, step=1, cv=StratifiedKFold(5), n_jobs=-1)
rfecv.fit(X_train, y_train)

print("Number of features selected: %d" % rfecv.n_features_)

Number of features selected: 9874

I then plot the number of features vs the CV scores:

plt.figure()
plt.xlabel("feature count")
plt.ylabel("CV accuracy")
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.show()

What I think is happening (and this is what I need an expert for) is that the first peak shows the optimal number of features. After that the curve drops and only starts to climb again because of overfitting, not really seperating classes but examples. Could this be the case? And if so how can I obtain these features (i.e. the ones at that first peak), because rfecv.support_ only gives me the ones where the highest accuracy was reached (meaning: all of them).

And while I am at it: How would I choose the best estimator for the RFE? Is it just by trial and error, going through all possible classifiers or is there any logic why I would use a Logit over a linear SVC for example?

Júlio Cesar Pereira Rocha Júlio Cesar Pereira Rocha · Accepted Answer · 2018-08-20T12:46:38

One way that i use for feature relevance is the RandomForest or ExtremeRandomizedTrees. i can use:

rfecv.n_features

to see how much features the find and:

 rfec.ranking

to see the features index in descending order. another algorithm that you can use is the PCA to reduce the dimension of you Dataset.

Recursive feature elemination with CV doesn't reduce feature count

1 Answers