Instead LBFGS, using gradient descent in sparse autoencoder

0
votes

In Andrew Ng's lecture notes, they use LBFGS and get some hidden features. Can I use gradient descent instead and produce the same hidden features? All the other parameters are the same, just change the optimization algorithm.

Because When I use LBFGS, my autoencoder can produce the same hidden features as in the lectures notes, but when I use gradient descent, the features in the hidden layer are gone, seems like totally random.

To be specific, in order to optimize the cost function, I implement 1)the cost function, 2)gradient of each Weight and Bias. And throw them into scipy optimize tool box to optimize the cost function. And this setting can give me the reasonable hidden features.

But when I change to gradient descent. I tried to let the "Weight - gradient of the Weight" and "Bias - gradient of the Bias". But the resulted hidden features looks like totally random.

Can somebody help me to know the reason? Thanks.

1
machine-learningmathematical-optimizationdeep-learninggradient-descentautoencoder
they use LBFGS and get some hidden features. Can I use gradient descent instead and produce the same hidden features? - in principle yes. At least if both converge. Gradient descent however can be painfully slow for some functions, so you may not end up in a local optimum in reasonable amount of time. Also the choice of the step-size will be critical if you want to implement the optimization yourself. - cel

1 Answers

1
votes

Yes, you can use SGD instead, in fact, it is the most popular choice in practise. L-BFGS-B is not a typical method for training neural networks. However:

  • you will have to tweak hyperparameters of the training method, you cannot just use the same ones that were used for LBFGS as this is completely different method (ok, not completely, but it uses first order optimization instead of second order)
  • you should include momentum in your SGD, it is an extremely easy way to get a kind of second order approximation, and is known to (when carefully tuned) perform as good as actual second-order methods in practise