I read some articles about restricted Boltzmann machines. These machines were tested for their reconstruction capabilities. I understand how training works, but not how this reconstruction is done. Can anybody give me some hints ?
4 Answers
I think you mean an RBM used in the context of an autoencoder for recalling inputs, rather like an associative memory network.
During the learning phase, the input units and output nodes (units) are clamped to values found in the training set, after which repeated Boltzmann sampling (e.g. via combination of Metropolis Hastings and Simulated Annealing) is used in conjunction wit gradient descent to optimize the weights over all connections between input, hidden and output nodes. After training, the restricted Boltzmann network can be used to either 1) classify/predict or 2) generate memory in free running mode.
In the classify/predict mode, some or all of the input units are clamped while the hidden and output nodes are sampled with Boltzmann sampling. The statistical properties (essentially the means) of the output nodes are the predictions or classifications. If the RBM is structured like a single layer autoencoder, with an input layer that resembles the output layer and a sparse or smaller hidden layer, then a limited number of inputs resulting in complete recovery of a training input would constitute a "reconstruction".
Hinton's 2006 Science paper discusses some of these ideas:
http://www.cs.toronto.edu/~hinton/science.pdf
This paper is a rather specific application of RBMs and neural networks though. In this paper he uses RBMs to pretrain a deep (many layered) neural network.
Restricted Boltzmann Machines are generative models that can generate samples for given hidden units.
For reconstruction, firstly, the input data is clamped to visible units and hidden states are calculated by using model's weight. In second step, the visible units are calculated by using recently calculated hidden states. Visible states that you get in second step are reconstructed sample.
Comparison of the input data and reconstructed sample(element-wise comparison etc.) gives reconstruction error.
In Hinton's algorithm, they use RBM for pre-training so that the initial weights are close to a good solution ensuring convergence of gradient descent.
In the next step, the initial network(say, [1000 500 100 2]) is unfolded to give an auto-encoder network([1000 500 100 2 100 500 1000]). The encoder([1000 500 100 2]) and decoder([2 100 500 1000]) parts initially use the same weights.
The last stage is fine-tuning. It uses backpropagation through the whole auto-encoder([1000 500 100 2 100 500 1000]) to fine-tune the weights(by minimizing the error which is the difference between input and its reconstruction) for optimal reconstruction. The output of auto-encoder is the reconstruction of the input.