Logistic regression error: New levels in categorical column in Test data

Question

As I was trying to predict using Logistic Regression model in R, I am having the following error:

Error

pred <- predict(model,newdata=test)
Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels) : factor admission_type_id has new levels 8

From my understanding, the testing data of the column "admission_type_id" has new levels compared training data. I tried looking at the unique values and I see that Train data has all the values of Test data.

unique(train$admission_type_id)
1 1 3 2 6 5 8 4
Levels: 1 2 3 4 5 6 7 8

unique(test$admission_type_id)
1 6 1 2 3 5 8
Levels: 1 2 3 4 5 6 7 8

It would be of great help if some one can help me understand the issue. Thank you.

Answer 1

The issue is that you are having different factor levels for one/many of the predictors:

For example, I am using mtcars data. It should also be present in your session.(Read my comments in between codes)

mtcars_train <- mtcars[mtcars$cyl %in% c(4,6),] #Consider this as train data
mtcars_train$cyl <- as.factor(mtcars_train$cyl) #Changing cyl to factor, here cyl values are 4 and 6 only, notice there is no 8

mtcars_orig <- mtcars #Taking the entire data which contain cyl values as 4,6 and 8
mtcars_orig$cyl <- as.factor(mtcars_orig$cyl) #Converting to factors, here again we can see that levels(mtcars_orign$cyl) is 4,6 and 8

lm1 <- lm(am ~ mpg + cyl + hp , data=mtcars_train) # Building the model
predict(lm1, newdata=mtcars_orig) #Now if you try to predict you will receive the error of having missing classes

lm1$xlevels[["cyl"]] <- union(lm1$xlevels[["cyl"]], levels(mtcars_orig$cyl)) #You can fix this here by running this code, it will append the levels to your original lm1 model here

After running the last step if you run the predict , you will have your outcome.

predict(lm1, newdata=mtcars_orig)

My suggestion is, take a train sample which is a very good representation of your entire dataset and your train and test should match, if there is a mismatch of factor levels this error will always come.