Random forest package in R shows error during prediction() if there are new factor levels present in test data. Is there any way to avoid this error?

4
votes

I have 30 factor levels of a predictor in my training data. I again have 30 factor levels of the same predictor in my test data but some levels are different. And randomForest does not predict unless the levels are same exactly. It shows error. Says, Error in predict.randomForest(model,test) New factor levels not present in the training data

4
rrandom-forest
Exactly how is it supposed to construct a prediction for something it has not 'seen' before? – IRTFM
A common workaround is to group the infrequent levels of the factor into an "other" level, that will also contain unobserved values. Another idea (if there are many variables) would be to discard the trees that use a variable with an unobserved value. – Vincent Zoonekynd

4 Answers

10
votes

One workaround I've found is to first convert the factor variables in your train and test sets into characters

test$factor <- as.character(test$factor)

Then add a column to each with a flag for test/train, i.e.

test$isTest <- rep(1,nrow(test))
train$isTest <- rep(0,nrow(train))

Then rbind them

fullSet <- rbind(test,train)

Then convert back to a factor

fullSet$factor <- as.factor(fullSet$factor)

This will ensure that both the test and train sets have the same levels. Then you can split back off:

test.new <- fullSet[fullSet$isTest==1,]
train.new <- fullSet[fullSet$isTest==0,]

and you can drop/NULL out the isTest column from each. Then you'll have sets with identical levels you can train and test on. There might be a more elegant solution, but this has worked for me in the past and you can write it into a little function if you need to repeat it often.

2
votes

Use this to make the levels match (here test and train refer to columns in the testing and training datasets)

test<-factor(test, levels=levels(train))
1
votes

simple solution to this would be rbind your test data with training data ,do prediction and subset the rbind data from predictions .Tested method

0
votes

This is the issue that occurs when the level of your test data doesn't match with the level of the training data.

A simple fix that will work:

  • load test data with character column as factors
  • then rbind() test data with train data
  • Now subset the test data rows from step 2 and go for the prediction