Feature Subset Selection

Question

Before we reduce the dimension of a data set, we apply a learning algorithm to that set, and we obtain an objective function which generates a result for a data sample. This may be our classifier or regressor.

After that, we apply feature selection or feature extraction approach. What I am wondering is about the subset selection algorithm ,which is the algorithm adapting feature selection approach :

According to resources I have read so far, "you start with an empty feature set, and in each step of the algorithm, the feature which increases performance of your objective function is selected and added to your feature set. This operation continues until adding new feature does not improve the performance of your classifier or regressor."

What if adding new feature continues to improve performance of my objective function ? In this case, I have to add all features to my feature set, which means I choose all features. However, I am trying to reduce the dimension of data samples.

Answer 1

It depends on the problem and your dataset; but in general, with the feature selection strategy that you are describing (Sequential Forward Selection) it is quite unlikely that the end result would be to keep all the variables. You are either going to find local minima or irrelevant variables in most cases.

However, in the rare case that this happens, this would be basically telling you that all the features in your dataset are important - i.e. removing any of them would harm the accuracy of your model.

If the above is not a problem for you, you can either modify your objective function (so it considers both the current accuracy and the % of features eliminated - maybe as a weighted objective) or change your feature selection heuristic (you can use, for example, Sequential Backward Selection - which is very similar but starts considering all the features initially and then tries to remove them one by one).