Missing values in cross validation resampling with Caret

Question

Im trying to use the train() function from the Caret package for fitting a K Nearest Neighbor model. This gives me an error. My code is:

 "%+%" <- function(x,y) paste(x, y, sep = "")
 set.seed(28)
 ContEnt <- trainControl(method = "repeatedcv", number = 10, repeats = 3)
 EducKnn <- train(as.formula("pp04b_cod ~ " %+% paste(VarEduc[!VarEduc %in% NoRel], collapse 
                = " + ")), EducPrueba, method = "knn", trctrl = ContEnt,
               tuneLength = 10)

This gives me in return:

Warning: predictions failed for Resample01: k= 5 Error in knn3Train(train = structure(c(0.569069692629571, 0.569069692629571,  : 
  unused argument (trctrl = list("cv", 10, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))

Many similar warning messeges, and at the end:

Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = 
trainInfo,  :
There were missing values in resampled performance measures.
Something is wrong; all the Accuracy metric values are missing:          
    Accuracy       Kappa    
 Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA  
 NA's   :10    NA's   :10   
Error: Stopping
>

Setting parameters to avoid cross validation, doesnt seems to fix it.

ContEnt <- trainControl(method = "none")

Also, setting na.action = na.omit in train() gives the same results. Interestingly, the knn() function of the class package, works flawlessly, with a .75 training set, over the same set of variables.

Entre <- createDataPartition(EducPrueba$pp04b_cod, 1, 0.75, list = FALSE)
EducKnn <- knn(train = EducPrueba[Entre, VarEduc[!VarEduc %in% NoRel]], test = EducPrueba[-Entre, 
VarEduc[!VarEduc %in% NoRel]], cl = EducPrueba$pp04b_cod[Entre], k = 5)

Finally, to be sure, EducPrueba has no NAs or NaNs:

> any(is.na(EducPrueba))
[1] FALSE
> any(unlist(lapply(EducPrueba, is.nan)))
[1] FALSE

The varibles in VarEduc are already centered and scaled. Does anyone know how to make it work? Im using R Portable 3.5.2 with RStudio. Packages caret 6.0-81 and class 7.3-15. I dont know how to upload the dataframe, so this can be a reproducible example.

Yes, indeed. Thanks for seeing that. I spend some time writing the question in a text file. That must have been replaced. Ive test it with trctrl = ContEnt, and the problem persist. — Santiago Capobianco

NelsonGon NelsonGon · Accepted Answer · 2019-01-29T15:22:55

Here is how to reproduce your error:

train(Species~.,data=iris,trctrl=trainControl(method="cv",numebr=5),
      metric="Accuracy",method="knn")
Something is wrong; all the Accuracy metric values are missing:
    Accuracy       Kappa    
 Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA  
 NA's   :3     NA's   :3    
>

Error: Stopping In addition: There were 50 or more warnings (use warnings() to see the first 50)

Here is the same model with the recommended changes:

train(Species~.,data=iris,trControl=trainControl(method="cv",number=5),
       metric="Accuracy",method="knn")
k-Nearest Neighbors 

150 samples
  4 predictor
  3 classes: 'setosa', 'versicolor', 'virginica' 

No pre-processing
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 120, 120, 120, 120, 120 
Resampling results across tuning parameters:

To solve your problem You need to change trctrl to trControl.

trainControl(method = "repeatedcv", number = 10, repeats = 3)
 EducKnn <- train(as.formula("pp04b_cod ~ " %+% paste(VarEduc[!VarEduc %in% NoRel], collapse 
                = " + ")), EducPrueba, method = "knn", trControl= ContEnt,
               tuneLength = 10)

Missing values in cross validation resampling with Caret

1 Answers