Query regarding next layer in a Stacked Denoising AutoEncoder

1
votes

So I am training a Stacked Denoising AutoEncoder with 3 layers per AutoEncoder. My goal is image classification through the use of stacked denoising autoencoders.

The method that I use to create the the image segments:

  • Segment entire image into 8x8 blocks
  • Resize the blocks into 1x64
  • Randomly select blocks (eg: mod 7 or something) and insert into TrainingMatrix

Steps to train autoencoder:

  • TrainingMatrix dimensions: [10000x64] (i.e. 10000 training samples of dimension 64)
  • 3 Layer System: [64 -> 32 -> 64]
  • Corrupt each training sample with gaussian noise
  • Train a neural net with Input: [corrupted input], Output: [uncorrupted data] & then BACKPROP

For the next DAE do I use the weights from the last layer (i.e. layer 2->3) to train the next layer? Or would I have to run another sample through the network and then use that to train the next layer?If we use the weights from layer 2->3 wouldn't we only have one sample set to train the next autoencoder? If this is the case then would the weights just be the randomly generated initial values of the Weighting matrix?

Final Steps after stacked DAE layers:

  • Run final DAE layer through a supervised layer such as an SVM

Sorry if this sounds like a trivial question.

1
machine-learningcomputer-visionneural-network

1 Answers

4
votes

My interpretation of the stacked denoising autoencoder is you train the first autoencoder (i.e. 64 -> 32 -> 64) with backprogogation and your noise-free input as the output as you would a typical neural network then push your data through the first layer into 32 dimensional space and run the same process (i.e. 32 -> 16 -> 32) and go forwards from there. You could then add another layer on. You theoretical could also do some sort of fine tuning on the network as you could also form a 64 -> 32 -> 16 -> 32 -> 64 network and fine tune the parameters, but that probably isn't necessary.

After those steps, you then take you input data in 64 dimensional space and push it through 64 -> 32 -> 16 -> your classifier. If you want to use a neural network as you classifier then you could continue with more layers after that and then run backprop on that all the way to the start thus achieving better results. The original work on stacked denoising autoencoders is here (PDF).

On a side note, if you are thinking of using an SVM, I believe that is referred to as learning the kernel, but I do not have the reference for that.