Example of 10-fold cross-validation with Neural network classification in MATLAB

Question

I am looking for an example of applying 10-fold cross-validation in neural network.I need something link answer of this question: Example of 10-fold SVM classification in MATLAB

I would like to classify all 3 classes while in the example only two classes were considered.

Edit: here is the code I wrote for iris example

load fisheriris                              %# load iris dataset

k=10;
cvFolds = crossvalind('Kfold', species, k);   %# get indices of 10-fold CV
net = feedforwardnet(10);


for i = 1:k                                  %# for each fold
    testIdx = (cvFolds == i);                %# get indices of test instances
    trainIdx = ~testIdx;                     %# get indices training instances

    %# train 

    net = train(net,meas(trainIdx,:)',species(trainIdx)');
    %# test 
    outputs = net(meas(trainIdx,:)');
    errors = gsubtract(species(trainIdx)',outputs);
    performance = perform(net,species(trainIdx)',outputs)
    figure, plotconfusion(species(trainIdx)',outputs)
end

error given by matlab:

Error using nntraining.setup>setupPerWorker (line 62)
Targets T{1,1} is not numeric or logical.

Error in nntraining.setup (line 43)
    [net,data,tr,err] = setupPerWorker(net,trainFcn,X,Xi,Ai,T,EW,enableConfigure);

Error in network/train (line 335)
[net,data,tr,err] = nntraining.setup(net,net.trainFcn,X,Xi,Ai,T,EW,enableConfigure,isComposite);

Error in Untitled (line 17)
    net = train(net,meas(trainIdx,:)',species(trainIdx)');

You need to supply some code first. At the very least show how you are implementing the neural network without cross validation and what parameter you are trying to tune. Also have a look at this answer: stackoverflow.com/a/28168462/1011724, all you need to change is the fun function — Dan
I dont have any code.Any DB like Iris is fine. Basically the answer you sent and the link in the question are both fine with SVM. However I dont know how to train and test NN classifier in Matlab. Any specification of classifier doesn't have any importance — Woeitg
Stack overflow is not a code writing service. It is a forum to ask for help when you are stuck with a coding problem. You need to attempt this yourself first! Have you read the docs on training a neural net in MATLAB? — Dan
Yes,Offcourse I tried it myself and I received either unexpected wrong answer or errors. — Woeitg
Your error is because if you look at species you will see it is a categorical variable (i.e. not numerical or logical). You need to break it up into 3 binary dummy variables — Dan

Dan Dan · Accepted Answer · 2016-01-14T07:56:30

It's a lot simpler to just use MATLAB's crossval function than to do it manually using crossvalind. Since you are just asking how to get the test "score" from cross-validation, as opposed to using it to choose an optimal parameter like for example the number of hidden nodes, your code will be as simple as this:

load fisheriris;

% // Split up species into 3 binary dummy variables
S = unique(species);
O = [];
for s = 1:numel(S)
    O(:,end+1) = strcmp(species, S{s});
end

% // Crossvalidation
vals = crossval(@(XTRAIN, YTRAIN, XTEST, YTEST)fun(XTRAIN, YTRAIN, XTEST, YTEST), meas, O);

All that remains is to write that function fun which takes in input and output training and test sets (all provided to it by the crossval function so you don't need to worry about splitting your data yourself), trains a neural net on the training set, tests it on the test set and then output a score using your preferred metric. So something like this:

function testval = fun(XTRAIN, YTRAIN, XTEST, YTEST)

    net = feedforwardnet(10);
    net = train(net, XTRAIN', YTRAIN');

    yNet = net(XTEST');
    %'// find which output (of the three dummy variables) has the highest probability
    [~,classNet] = max(yNet',[],2);

    %// convert YTEST into a format that can be compared with classNet
    [~,classTest] = find(YTEST);


    %'// Check the success of the classifier
    cp = classperf(classTest, classNet);
    testval = cp.CorrectRate; %// replace this with your preferred metric

end

I don't have the neural network toolbox so I am unable to test this I'm afraid. But it should demonstrate the principle.

Example of 10-fold cross-validation with Neural network classification in MATLAB

1 Answers