Neural Network with Dropout

Question

I'm trying to use neural network for classification, the dataset consists of 7 parameters and more than 300k samples. By using dropout, i would like to see what is the effect of dropout to my network.I am using Matlab. However, matlab does not have a complete documentation for dropout. I found some papers about dropout, but most of them are for deep neural network for image and has a very complex approach. I have try to use nnstart by matlab, however i am not sure i can add dropout this way. Is there any way to use dropout by matlab for simple neural network?

It would be great if you could share some information or possibly a matlab code that is related to my question.

Answer 1

You will need to hold out a validation set from your 300K samples. Then train and test a net on the other samples and evaluate performance on the held-out validation set. Do this for your dropout AND non-dropout networks.

Answer 2

The idea behind using dropouts is to prevent overfitting. If you set dropout to 0.1, then for each iteration within each epoch, each node in that layer has a 10% probability of being dropped from the neural network. This essentially forces the network to learn deeper and more important relationships, rather than learning trivial relationships that lead to overfitting.

A good way is to imagine this is by picturing a neural network based binary classifier that predicts whether the image is of a cat or not. If we use images of orange colored cats while training without dropouts, the models ability to detect whether the image of a black colored cat is actually a cat might be very low. However, when you add dropouts, it may begin to understand the characteristic features of a cat rather than the characteristic features of an orange cat because fewer (and more important) nodes will be activated, which are probably the more important ones.

Although I haven't used Matlab before, you could probably create a custom function in matlab that follows the pseudocode below:

go through each of the nodes:
   if random(0,10) == 1:
      drop/turn off node i