Error when using varImp for nnet neural network

Question

I need to use varImp function on neural network model created by nnet method via caret.

The code:

#Load Packages
require(quantmod)
require(nnet)
require(caret)

#Creating data
T <- seq(0,20,length=200)                                   
y <- 1 + 3*cos(4*T+2) +.2*T^2 + rnorm(200)                  
dat <- data.frame( y, x1=Lag(y,1), x2=Lag(y,2),x3=Lag(y,3)) 
dat <- dat[4:length(dat[,1]),]
names(dat) <- c('y','x1','x2','x3')

set.seed(100) 
podzial <- createDataPartition(y, p = 3/4, list = FALSE)
zucz<-dat[podzial,]
ztest<-dat[-podzial,]

# Train control
ctrl <- trainControl(method = "LGOCV",p=0.7)

#Training
model <- train(y ~ x1+x2 , zucz, method='nnet', linout=TRUE, trace=F,maxit=100,skip=T,
           tuneGrid=expand.grid(.size=c(10),.decay=c(0,0.1,0.001,0.0001)),
           trControl = ctrl,
           preProcess = c("range"))

varImp(model,scale=F)

When I try to use varImp the error occurs:

Error in i2h[hidden, input] <- abeta[grep(label, nms, fixed = TRUE)] : 
number of items to replace is not a multiple of replacement length

I have done several test with diferent number of neurons. It appears that error occurs when number of neurons (size parameter) is bigger than 9. How to fix it?

This means that the total length ( hidden * input) of the output "region" is not a match for the length of what abeta is putting out. You'll need to fix that. — Carl Witthoft

MrFlick MrFlick · Accepted Answer · 2014-05-26T18:05:56

I think this is very clearly a bug in the caret package, specifically in the caret:::GarsonWeights function. This function tries to decode the coefficient names from your final model. They are

names(coef(model$finalModel))
#  [1] "b->h1"   "i1->h1"  "i2->h1"  "b->h2"   "i1->h2"  "i2->h2"  "b->h3"  
#  [8] "i1->h3"  "i2->h3"  "b->h4"   "i1->h4"  "i2->h4"  "b->h5"   "i1->h5" 
# [15] "i2->h5"  "b->h6"   "i1->h6"  "i2->h6"  "b->h7"   "i1->h7"  "i2->h7" 
# [22] "b->h8"   "i1->h8"  "i2->h8"  "b->h9"   "i1->h9"  "i2->h9"  "b->h10" 
# [29] "i1->h10" "i2->h10" "b->o"    "h1->o"   "h2->o"   "h3->o"   "h4->o"  
# [36] "h5->o"   "h6->o"   "h7->o"   "h8->o"   "h9->o"   "h10->o"

Here you have 2 input and 10 hidden nodes. The function tries to find the coefficent for input 1 and hidden note 1 by grepping for "i1->h1" with

grep("i1->h1",names(coef(model$finalModel)),fixed=T)
# [1]  2 29

But as you can see, two columns are returned, both "i1->h1" and "i1->h10". This is why the problem only happens when you get above 9 nodes. So clearly grep is not the correct function here. This is what's causing the error you see.

Now, how to fix it. First, we need a corrected version of GarsonWeights. So what i'm going to do, is copy the version from caret and attempt to replace the grep with match

gw <- caret:::GarsonWeights
body(gw)[[c(7,4,2,4,3,3,3)]] <- quote(match(label,nms))
body(gw)[[c(8,4,2,4,3,3,3)]] <- quote(match(label,nms))

Here i'm actually digging into the body of the function and changing just one section. Now this method for fixing a function is very fragile and may change with different versions. I tested this using caret_6.0-21.

The second challenge is to get varImp to use this updated version of the weights function. Unfortunately we can't directly update the one in the caret namespace and the default code explicitly uses that one. So what we're going to need to do, is actually change the model object which tells varImp() which how to run. If you look at model$modelInfo$varImp you can see where it's calling caret:::GarsonWeights. We just need to change that to our gw function. We can do that with

body(model$modelInfo$varImp)[[c(2,3)]]<-quote(gw(object,...))

I think this should work. You will need to re-run on every model you get when you run train() with method="nnet".

I suggest you contact the caret maintainer and report this bug. Feel free to provide a link to this answer for a full description.

Error when using varImp for nnet neural network

1 Answers