encoding categorical features?

Question

I have used one-hot-encoding to encode the categorical features in my dataset. After encoding, I found out that the training data results in more features than the testing data.

Example

A feature called week-day has 1,2,3,4,5 values and the one-hot-encoding results into 5 new features as expected. However the testing data of the same feature has values of 1,2 and result in 2 new features of one hot encoded.

My question

The training input has more features than the test data, only two inputs. How does this affect the model? Does the model works after all? Because of less input in the test data, how it will handle it?

Any help or suggestion is highly appreciated.

You'll end up testing the model on biased data that isn't as rich as the real data since it's missing some categories. Thus, you won't be able to tell if the model performs poorly on these categories. Why not randomly split the whole dataset into training ant test? — ForceBru
That is actually a good idea, but to prevent data leakage, the training and test datas are given in different files. — abanfo

amdex amdex · Accepted Answer · 2020-12-04T15:01:34

This is not a problem, per sé, although a standard assumption in most ML settings is that the label distribution of the test set is identical to that of the training set.

You should just re-encode your test data using the same features as you would encode your training data. That is, any features from test that also occur in train should get the same index those features got in test, while any features occurring in test that do not occur in train should be discarded (because it is unclear what you should do with them anyway).

encoding categorical features?

1 Answers