Keras - Variational Autoencoder NaN loss

4
votes

I'm trying to use the implementation of Variational Autoencoder that I found among the Keras examples (https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py).

I just refactored the code in order to use it more easily from a Jupyter notebook (my code: https://github.com/matbell/Autoencoders/blob/master/models/vae.py).

However, when I try to fit the model on my data I get the following output:

Autoencoders/models/vae.py:69: UserWarning: Output "dense_5" missing from loss dictionary. We assume this was done on purpose, and we will not be expecting any data to be passed to "dense_5" during training.
self.vae.compile(optimizer='rmsprop')

Train on 15474 samples, validate on 3869 samples
Epoch 1/50
15474/15474 [==============================] - 1s 76us/step - loss: nan - val_loss: nan
Epoch 2/50
15474/15474 [==============================] - 1s 65us/step - loss: nan - val_loss: nan
Epoch 3/50
15474/15474 [==============================] - 1s 69us/step - loss: nan - val_loss: nan
Epoch 4/50
15474/15474 [==============================] - 1s 62us/step - loss: nan - val_loss: nan

and the loss remains the same for all the training epochs.

I'm not so expert in Deep Learning and Neural Networks fields, so maybe I'm missing something....

This is the input data, where data and labels are two pandas.DataFrame.

In: data.shape
Out: (19343, 87)

In: label.shape
Out: (19343, 1)

And this is how I use the Vae class (from my code) in Jupyter notebook:

INPUT_SIZE = len(data.columns)
X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size = 0.2)

vae = Vae(INPUT_SIZE, intermediate_dim=32)
vae.fit(X_train, X_test)

Thanks for any help!

5
neural-networkdeep-learningkerasautoencoder
can you add the changes you made to the original code so that we can trace the error you would have made more easily? - Shashi Tunga
@ShashiTunga I reported the links of both original code and my "modification". As you can note, I didn't make any changes to the code, I just reformatted it as a Python class with three main methods: init(), fit(), and encode(). - Mattia Campana
Which version of Keras did you use? - kvorobiev

5 Answers

9
votes

You might want to initialize your log_var dense layer to zeros. I was having problems with it myself (slightly different code, but effectively doing the same), and it turns out that, however small the variation weights were initialized to, they would explode in just a few rounds of SGD.

The random correlations between epsilon ~N(0,1) and the reconstruction error will be enough to gently bring the weights to nonzero.

Edit - also, the exponential wrapping the variation really helps exploding the gradients. Setting the initial value of the weights to zero gives an initial variation of one, because of the exponential. Initializing it to a low negative value, while giving off an initial close-to-zero variation, makes the gradient enormous on the very first runs. Zero gives me the best results.

5
votes

If none of the above helped you. Make sure your inputs are normalize between [0, 1]. An easy way to do it with MNIST is

X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
4
votes

Your input consist of NaN values which is why you are seeing the Nan in output. You can count NaN in your numpy array using the following:

np.count_nonzero(np.isnan(data))

If the NAN are not relevant, just remove them from your training data or map them to a specific constant (like 0 or -1) which will take care of your problem

1
votes

I had a similar problem because there were inputs that contained NaNs. If there are just a few examples with a NaN in there, all the weights and losses will be NaN too. Check the content of the data again!

0
votes

You fit the model incorrectly. Your fit call implies: vae.fit(X = X_train, y = X_test) , while it should be: 

vae.fit(X_train,X_train , validation_data = X_test)