Example Request: unsupervised deep learning in python

Question

Context

I'm relatively new to neural nets and would like to learn about clustering methods that are able to make class predictions after learning a representation.

Some tutorials online for autoencoders/rbms/deep belief networks typically have a supervised fit() call such as fit(X,y) or Pipeline(rbm, logistic). See: http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/

http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html

I'd like to explore the effect of hidden layers on unlabeled data, so algorithms like k-means won't quite suffice.

Request

It would be nice to see a Python example that has calls similar to fit(X) and predict(Y), where X and Y are unlabeled datasets. The idea is that predict() operates by finding a 'closest' class as determined by the representation learned in fit().

I certainly don't expect a full implementation, but relevant resources would be appreciated.

For example, in http://deeplearning.net/tutorial/DBN.html, it seems we can build a DBN. Is there a corresponding predict() method?

Addenda

A somewhat related question:

Getting the learned representation of the data from the unsupervised learning in pylearn2

Answer 1

In python deep learning packages you usually have to first define the architecture of your model and then train (fit) it.

The simplest application of Auto-Encoders I can think of is in keras

You first need to define the size of the hidden (compressed) representation.

hidden_dim = 32

Make the necessary imports

from keras.layers import Input, Dense
from keras.models import Model

Then define your model's architecture

input = Input(shape=(xxx,))
encode = Dense(hidden_dim, activation='relu')(input)
decode = Dense(xxx, activation='sigmoid')(encode)

autoencoder = Model(input, decode)

The xxx above is the dimension of your input. For example if you want to train the autoencoder on the MNIST dataset (which has 28x28 images), xxx would be 28x28=784.

Now compile your model with the cost function and the optimizer of your choosing

autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

Now to train your unsupervised model, you should place the same image as the input and the output. Some approaches also add noise to the output but I'll leave that up to you. Assuming that X is your training data and X_val is your validation data. Adjust the hyperparameters according to your needs.

autoencoder.fit(X, X, epochs=100, batch_size=32, shuffle=True, validation_data=(X_val, X_val)

Then say you have a test set called X_test, you can ask your model to try and reproduce it.

y_hat = autoencoder.predict(X_test)

TL;DR
It is a bit more hard to do than with sklearn but the basic steps are:

• Define your network architecture (layers, activations, shapes, etc.)
• Compile your model (define cost function and optimizer)
• Fit your model on your data (also define training parameters)
• Predict output given test input.

Getting the internal representation

In order to now answer your second question you need to define the encoder and decoder separately

 encoder = Model(input, encode)
 encoded_input = Input(shape=(hidden_dim,))
 decoder_layer = autoencoder.layers[-1]
 decoder = Model(encoded_input, decoder_layer(encoded_input))

Now just pass a test input (let's call it original) through your encoder and decoder

hidden_representation = encoder.predict(original)
recreation = decoder.predict(hidden_representation)

You can also use the hidden representation or even the encoder layer to generate the inputs of another algorithm (for example a supervised one)