Calculate Prediction Intervals of a predicted value using Caret package of R

Question

I used different neural network packages within Caret package for my predictions. Code used with nnet package is

    library(caret)
    # training model using nnet method
    data <- na.omit(data)
    xtrain <- data[,c("temperature","prevday1","prevday2","prev_instant1","prev_instant2","prev_2_hour")]
    ytrain <- data$power
    train_model <- train(x = xtrain, y = ytrain, method = "nnet", linout=TRUE, na.action = na.exclude,trace=FALSE)
    # prediction using training model created
    pred_ob <- predict(train_model, newdata=dframe,type="raw")

The predict function simply calculates the prediction value. But, I also need prediction intervals (2-sigma) as well. On searching, I found a relevant answer at stackoverflow link, but this does not result as needed. The solution suggests to use finalModelvariable as

predict(train_model$finalModel, newdata=dframe, interval = "confidence",type=raw)

Is there any other way to calculate prediction intervals? The training data used is the dput() of my previous question at stackoverflow link and the dput() of my prediction dataframe (test data) is

    dframe <- structure(list(temperature = 27, prevday1 = 1607.69296666667, 
    prevday2 = 1766.18103333333, prev_instant1 = 1717.19306666667, 
    prev_instant2 = 1577.168915, prev_2_hour = 1370.14983583333), .Names = c("temperature", 
"prevday1", "prevday2", "prev_instant1", "prev_instant2", "prev_2_hour"
), class = "data.frame", row.names = c(NA, -1L))

****************************UPDATE***********************

I used nnetpredintpackage as suggested at link. To my surprise it results in an error, which I find difficult to debug. Here is my updated code till now,

library(nnetpredint)
nnetPredInt(train_model, xTrain = xtrain, yTrain = ytrain,newData = dframe)

It results in the following error:

Error: Number of observations for xTrain, yTrain, yFit are not the same
[1] 0

I can check that xtrain, ytrain and dframe are with correct dimensions, but I do not have any idea about yFit. I don't need this according to the examples of nnetpredintvignette

Answer 1

caret doesn't generate prediction intervals; that relies on the individual package. If that package cannot do this, then neither can the train objects. I agree that nnetPredInt is the appropriate way to go.

Two other notes:

• you most likely should center and scale your data if you have not already.
• using the finalModel object is somewhat dangerous since it has no idea what was done to the data (e.g. dummy variables, centering and scale or other preprocessing methods, etc) before it was created.

Max

Answer 2

Thanks for your question. And a simple answer to your problem is: Right now the nnetPredInt function only support the following S3 object, "nnet", "nn" and "rsnns", produced by different neural network packages. And the train function in caret package return an "train" object. That's why the function nnetPredInt doesn't get the yFit vectors, which is the fitted.value of the training datasets, from your train_model.

1.Quick way to use the model from caret package: Get the finalModel result from the 'train' object:

    nnetObj = train_model$finalModel # return the 'nnet' model which the caret package has found.
    yPredInt = nnetPredInt(nnetObj, xTrain = xtrain, yTrain = ytrain,newData = dframe)

For Example, Use the Iris Dataset and the 'nnet' method from caret package for regression prediction.

    library(caret)
    library(nnetpredint)
    # Setosa 0 and Versicolor 1
    ird <- data.frame(rbind(iris3[,,1], iris3[,,2]), species = c(rep(0, 50), rep(1, 50)))
    samp = sample(1:100, 80)
    xtrain = ird[samp,][1:4]
    ytrain = ird[samp,]$species
    # Training
    train_model <- train(x = xtrain, y = ytrain, method = "nnet", linout = FALSE, na.action = na.exclude,trace=FALSE)
    class(train_model) # [1] "train"

    nnetObj = train_model$finalModel
    class(nnetObj) # [1] "nnet.formula" "nnet" 

    # Constructing Prediction Interval
    xtest = ird[-samp,][1:4]
    ytest = ird[-samp,]$species
    yPredInt = nnetPredInt(nnetObj, xTrain = xtrain, yTrain = ytrain,newData = xtest)

    # Compare Results: ytest and yPredInt
    ytest
    yPredInt

2.The Hard Way

Use the generic nnetPredInt function to pass all the neural net specific parameters to the function:

    nnetPredInt(object = NULL, xTrain, yTrain, yFit, node, wts, newData,alpha = 0.05 , lambda = 0.5, funName = 'sigmoid', ...)

    xTrain # Training Dataset
    yTrain # Training Target Value
    yFit # Fitted Value of the training data

    node # Structure of your network, like c(4,5,5,1)
    wts # Specific order of weights parameters found by your neural network
    newData # New Data for prediction

Tips: Right now nnetpredint package only support the standard multilayer neural network regression with activated output, not the linear output, And it will support more type of models soon in the future.

Answer 3

You can use the nnetPredInt function {package:nnetpredint}. Check out the function's help page here

Answer 4

If you are open to writing your own implementation there is another option. You can get prediction intervals from a trained net using the same implementation you would write for standard non-linear regression (assuming back propagation was used to do the estimation).

This paper goes through the methodology and is fairly straight foward: http://www.cis.upenn.edu/~ungar/Datamining/Publications/yale.pdf.

There are, as with everything,some cons (outlined in the paper) to this approach but definitely worth knowing as an option.