Is the training method of a Convolutional Network still known as deep learning?

4
votes

In papers such as ImageNet Classification with Deep Convolutional Neural Networks

http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

the training method seems to be basic backpropagation with stochastic gradient descent.

Even though CNNs are part of deep neural networks, is this purely because of the large number of hidden layers present? And does this mean that the backprop here falls under the category of deep learning because the network is deep, even though it does not follow the same pattern as the likes of a DBN using greedy layer wise training, a true deep learning technique?

Thanks for the help and advice.

3
machine-learningartificial-intelligenceneural-networkconvolutiondeep-learning
This is called deep due to number of hidden layers are more than its ancestors. They are successful since people find more ways to deal diminishing gradient problem for deeper NN models (ReLU, Dropout, Maxout, Response Normalization...). They are viable due to GPUs. - erogol
@Erogol Do you know any recent summary paper that describe those techniques that address the diminishing gradient problem? - ziggystar

3 Answers

2
votes

If you read the Wikipedia page of Deep Learning it said: "Deep Learning is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using multiple processing layers, with complex structures or otherwise, composed of multiple non-linear transformations".

A CNN has multiple layers of non-linear transformation, so qualifies as a Deep Learning model.

Also in this book from MIT http://www.deeplearningbook.org/ CNN are also part of Deep Learning models.

Between DBN and CNN exist an important difference, the first is an unsupervised model and the other isn't, besides one use DBN for pre-initialization.

If you read about RNN or LSTM that are also Deep Learning models you will find that both are basically trained with a modified version of backpropagation called backpropagation through time.

So, remember the concept multiple non-linear transformations for model high-level abstractions in data.

Also Deep Learning refers to the model not to the training.

0
votes

according to my knowledge

1 neuron and 1 layer -> known as a perceptron

2 or more neurons, 1 layer, 1 input --> as an equation of a line y = w*x + w0

2 or more neurons, 1 layer, multiple input instances --> hyper plane

when you add the sigmoid functionality to the output of these neurons activations and combine these in another perceptron or multiple neurons you get a non-linear combination of hyper-planes.

the thing that make CNN to be called CNN instead of a simple Deep Neural Network is that you learn local neighborhood weights that can be in any part of the image. So there is a weight sharing process between neuron activations.

  • suppose you had N different 20x20 gray scale images and 5 hidden units in 1 layer,
  • if you were to implement a fully connected deep neural network, you would try learning a weight matrix of 400x5 - hence 2000 parameters - for the first hidden layer. and as the output you would have 5 dimensional vectors for each N image.
  • but in a cnn structure, you decide on a reasonable patch size in these 20x20 images, lets say 4x4, and learn 5 different 4x4 weights - hence 80 parameters. And as the output of this first hidden layer you would have 5 different 17x17 images for each N image. In other way of looking at the output of first layer is 1445 dimensional vectors for each N image. So it is like you learn less parameters but as the output you have more dimensions to learn from.

So when I look at your questions,

  1. If there was only 1 layer and the classification/regression was made after that 1 single layer it wouldn't be called DBN. But it could still be called CNN because there is the concept of 'convolution' and 'neural network'. And of course with 1 single layer there would be no 'backpropogation'.

  2. Backpropogation is needed when there is more than 1 layer present. Because the concept is to be able to backpropogate some error without having a 'real/expected output' which we can see in the middle layers. We only have ground truth for the last layer where we have our regression/classification. Hence, if I understood your question correctly, backprop here falls under deep network category.

  3. We can not say 'CNNs do not follow same pattern as DBN'. Because they certainly are DBNs with weight sharing functionalities.

I hope this answers your questions. And additionally I think it would be nice to point out the difference between DBN and DNN here.

Quoting from another website (https://www.quora.com/Deep-Learning/What-are-the-difference-and-relevance-between-DNN-and-DBN)

“Deep Belief Networks construct beliefs (probablilistical relationships between instances) based on unsupervised data, and then apply those relationships to a problem when presented with supervised data.

Basically, learn on unsupervised, where there is more data to learn from, then use on a problem.

Deep neural networks are simply large multilayer neural networks.”

0
votes

deep learning techniques are recent technology for Artificial Intelligence in specific Convolutional neural networks(CNN) are very effective in pattern recognition,object or face recognition many libraries are available for CNN like Itorch ,theano,Digits etc for deep understanding of neural networks and deep learning click here