Deep autoencoder keeping constant accuracy in keras

2
votes

I'm new in keras and deep learning field. In fact, I built a deep autoencoder using keras library based on ionosphere data set, which contains a mixed data frame (float, strings"objects", integers..) so I tried to replace all object colunms to float or integer type since the autoencoder refuses being fed with object samples. The training set contains 10000 samples with 48 columns and the validation set contains 5000 samples. I didn't do any normalization of the input data because I thought isn't really necessary for autoencoder model. I used a binary cross entropy loss function, am not sure if that could be the reason of having a constant loss and a constant accuracy values during training. I tried different number of epochs but I got the same thing. I tried also to change the batch size but no change. Can any one help me find the problem please.

input_size = 48
hidden_size1 = 30
hidden_size2 = 20
code_size = 10
batch_size = 80

checkpointer = ModelCheckpoint(filepath="model.h5",
                               verbose=0,
                               save_best_only=True)
tensorboard = TensorBoard(log_dir='./logs',
                          histogram_freq=0,
                          write_graph=True,
                          write_images=True)
input_enc = Input(shape=(input_size,))
hidden_1 = Dense(hidden_size1, activation='relu')(input_enc)
hidden_11 = Dense(hidden_size2, activation='relu')(hidden_1)
code = Dense(code_size, activation='relu')(hidden_11)
hidden_22 = Dense(hidden_size2, activation='relu')(code)
hidden_2 = Dense(hidden_size1, activation='relu')(hidden_22)
output_enc = Dense(input_size, activation='sigmoid')(hidden_2)
autoencoder_yes = Model(input_enc, output_enc)

autoencoder_yes.compile(optimizer='adam',
                         loss='binary_crossentropy', 
                         metrics=['accuracy'])
history = autoencoder_yes.fit(df_noyau_yes, df_noyau_yes,
                               epochs=200,
                                batch_size=batch_size,
                                shuffle = True,
                                validation_data=(df_test_yes, df_test_yes),
                                verbose=1, 
                                callbacks=[checkpointer, tensorboard]).history

Epoch 176/200

  80/7412 [..............................] - ETA: 2s - loss: -15302256.0000 - acc: 0.4357
 320/7412 [>.............................] - ETA: 2s - loss: -16773740.2500 - acc: 0.4448
 480/7412 [>.............................] - ETA: 2s - loss: -16924116.1667 - acc: 0.4444
 720/7412 [=>............................] - ETA: 2s - loss: -17179484.1111 - acc: 0.4460
 960/7412 [==>...........................] - ETA: 2s - loss: -17382038.5833 - acc: 0.4463
1120/7412 [===>..........................] - ETA: 1s - loss: -17477103.7857 - acc: 0.4466
1360/7412 [====>.........................] - ETA: 1s - loss: -17510808.8824 - acc: 0.4466
1520/7412 [=====>........................] - ETA: 1s - loss: -17337536.3158 - acc: 0.4462
1680/7412 [=====>........................] - ETA: 1s - loss: -17221621.6190 - acc: 0.4466
1840/7412 [======>.......................] - ETA: 1s - loss: -17234479.0870 - acc: 0.4467
2000/7412 [=======>......................] - ETA: 1s - loss: -17336909.4000 - acc: 0.4469
2160/7412 [=======>......................] - ETA: 1s - loss: -17338357.2222 - acc: 0.4467
2320/7412 [========>.....................] - ETA: 1s - loss: -17434196.3103 - acc: 0.4465
2560/7412 [=========>....................] - ETA: 1s - loss: -17306412.6875 - acc: 0.4463
2720/7412 [==========>...................] - ETA: 1s - loss: -17229429.4118 - acc: 0.4463
2880/7412 [==========>...................] - ETA: 1s - loss: -17292270.6667 - acc: 0.4461
3040/7412 [===========>..................] - ETA: 1s - loss: -17348734.3684 - acc: 0.4463
3200/7412 [===========>..................] - ETA: 1s - loss: -17343675.9750 - acc: 0.4461
3360/7412 [============>.................] - ETA: 1s - loss: -17276183.1429 - acc: 0.4461
3520/7412 [=============>................] - ETA: 1s - loss: -17222447.5455 - acc: 0.4463
3680/7412 [=============>................] - ETA: 1s - loss: -17179892.1304 - acc: 0.4463
3840/7412 [==============>...............] - ETA: 1s - loss: -17118994.1667 - acc: 0.4462
4080/7412 [===============>..............] - ETA: 1s - loss: -17064828.6275 - acc: 0.4461
4320/7412 [================>.............] - ETA: 0s - loss: -16997390.4074 - acc: 0.4460
4480/7412 [=================>............] - ETA: 0s - loss: -17022740.0357 - acc: 0.4461
4640/7412 [=================>............] - ETA: 0s - loss: -17008629.1552 - acc: 0.4460
4880/7412 [==================>...........] - ETA: 0s - loss: -16969480.9836 - acc: 0.4459
5040/7412 [===================>..........] - ETA: 0s - loss: -17028253.4921 - acc: 0.4457
5200/7412 [====================>.........] - ETA: 0s - loss: -17035566.0308 - acc: 0.4456
5360/7412 [====================>.........] - ETA: 0s - loss: -17057620.4776 - acc: 0.4456
5600/7412 [=====================>........] - ETA: 0s - loss: -17115849.8857 - acc: 0.4457
5760/7412 [======================>.......] - ETA: 0s - loss: -17117196.7500 - acc: 0.4458
5920/7412 [======================>.......] - ETA: 0s - loss: -17071744.5676 - acc: 0.4458
6080/7412 [=======================>......] - ETA: 0s - loss: -17073121.6184 - acc: 0.4459
6320/7412 [========================>.....] - ETA: 0s - loss: -17075835.3797 - acc: 0.4461
6560/7412 [=========================>....] - ETA: 0s - loss: -17081048.5610 - acc: 0.4460
6800/7412 [==========================>...] - ETA: 0s - loss: -17109489.2471 - acc: 0.4460
7040/7412 [===========================>..] - ETA: 0s - loss: -17022715.4545 - acc: 0.4460
7200/7412 [============================>.] - ETA: 0s - loss: -17038501.4222 - acc: 0.4460
7360/7412 [============================>.] - ETA: 0s - loss: -17041619.7174 - acc: 0.4461
7412/7412 [==============================] - 3s 357us/step - loss: -17015624.9390 - acc: 0.4462 - val_loss: -26101260.3556 - val_acc: 0.4473

Epoch 200/200

  80/7412 [..............................] - ETA: 2s - loss: -16431795.0000 - acc: 0.4367
 240/7412 [..............................] - ETA: 2s - loss: -16439401.0000 - acc: 0.4455
 480/7412 [>.............................] - ETA: 2s - loss: -16591146.5000 - acc: 0.4454
 640/7412 [=>............................] - ETA: 2s - loss: -16914542.8750 - acc: 0.4457
 880/7412 [==>...........................] - ETA: 2s - loss: -16552313.2727 - acc: 0.4460
1120/7412 [===>..........................] - ETA: 1s - loss: -16839956.4286 - acc: 0.4459
1280/7412 [====>.........................] - ETA: 1s - loss: -16965753.3750 - acc: 0.4461
1440/7412 [====>.........................] - ETA: 1s - loss: -17060988.4444 - acc: 0.4461
1680/7412 [=====>........................] - ETA: 1s - loss: -17149844.2381 - acc: 0.4462
1840/7412 [======>.......................] - ETA: 1s - loss: -17049971.6957 - acc: 0.4462
2080/7412 [=======>......................] - ETA: 1s - loss: -17174574.2692 - acc: 0.4462
2240/7412 [========>.....................] - ETA: 1s - loss: -17131009.5357 - acc: 0.4463
2480/7412 [=========>....................] - ETA: 1s - loss: -17182876.8065 - acc: 0.4461
2720/7412 [==========>...................] - ETA: 1s - loss: -17115984.6176 - acc: 0.4460
2880/7412 [==========>...................] - ETA: 1s - loss: -17115818.8611 - acc: 0.4459
3120/7412 [===========>..................] - ETA: 1s - loss: -17123591.0256 - acc: 0.4459
3280/7412 [============>.................] - ETA: 1s - loss: -17114971.6585 - acc: 0.4459
3440/7412 [============>.................] - ETA: 1s - loss: -17072177.0698 - acc: 0.4462
3600/7412 [=============>................] - ETA: 1s - loss: -17025446.1333 - acc: 0.4460
3840/7412 [==============>...............] - ETA: 1s - loss: -16969630.0625 - acc: 0.4462
4080/7412 [===============>..............] - ETA: 1s - loss: -16961362.9608 - acc: 0.4461
4320/7412 [================>.............] - ETA: 0s - loss: -16969639.5000 - acc: 0.4461
4480/7412 [=================>............] - ETA: 0s - loss: -16946814.6964 - acc: 0.4462
4640/7412 [=================>............] - ETA: 0s - loss: -16941803.2586 - acc: 0.4461
4880/7412 [==================>...........] - ETA: 0s - loss: -16915578.2623 - acc: 0.4462
5040/7412 [===================>..........] - ETA: 0s - loss: -16916479.5714 - acc: 0.4464
5200/7412 [====================>.........] - ETA: 0s - loss: -16896774.5846 - acc: 0.4463
5360/7412 [====================>.........] - ETA: 0s - loss: -16956822.5075 - acc: 0.4462
5600/7412 [=====================>........] - ETA: 0s - loss: -17015829.3286 - acc: 0.4461
5760/7412 [======================>.......] - ETA: 0s - loss: -17024089.8750 - acc: 0.4460
5920/7412 [======================>.......] - ETA: 0s - loss: -17034422.1216 - acc: 0.4462
6160/7412 [=======================>......] - ETA: 0s - loss: -17042738.7273 - acc: 0.4462
6320/7412 [========================>.....] - ETA: 0s - loss: -17041053.0886 - acc: 0.4462
6480/7412 [=========================>....] - ETA: 0s - loss: -17046979.9012 - acc: 0.4461
6640/7412 [=========================>....] - ETA: 0s - loss: -17041165.7590 - acc: 0.4461
6800/7412 [==========================>...] - ETA: 0s - loss: -17070702.2824 - acc: 0.4460
7040/7412 [===========================>..] - ETA: 0s - loss: -17031330.6364 - acc: 0.4460
7280/7412 [============================>.] - ETA: 0s - loss: -17027056.8132 - acc: 0.4461
7412/7412 [==============================] - 3s 363us/step - loss: -17015624.9908 - acc: 0.4462 - val_loss: -26101260.3556 - val_acc: 0.4473

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 48)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 30)                1470      
_________________________________________________________________
dense_2 (Dense)              (None, 20)                620       
_________________________________________________________________
dense_3 (Dense)              (None, 10)                210       
_________________________________________________________________
dense_4 (Dense)              (None, 20)                220       
_________________________________________________________________
dense_5 (Dense)              (None, 30)                630       
_________________________________________________________________
dense_6 (Dense)              (None, 48)                1488      
=================================================================
Total params: 4,638
Trainable params: 4,638
Non-trainable params: 0
_________________________________________________________________
None
2
pythonmachine-learningdeep-learningkerasautoencoder

2 Answers

2
votes

You may already have fixed your issue, but I want to clarify what probably went wrong with your autoencoder so other people that have the same problem can understand what is happening.

The main issue is that you didn't normalize your input data and you used sigmoid function as activation in the last layer.

What this means is that you have an input that can range between -infinity and +infinity while your output data can only vary between 0 and 1.

An autoencoder is a Neural Network that tries to learn the identity function. What this means is that if you have an input [0, 1, 2, 3], you want the network to output [0, 1, 2, 3].

What happened in your case is that you used sigmoid as an Activation Function in the last layer, which means that every value this layer receives, it will apply a sigmoid function.

As said before, the sigmoid function squashes values between 0 and 1. So if you have [0, 1, 2, 3] as inputs, even if your hidden layer learns the identity function (which I think would be impossible in this case), the output would be sigmoid([0, 1, 2, 3]), which results in [0.5, 0.73, 0.88, 0.95] approximately.

If you think about it, it's impossible that this autoencoder learns to replicate its inputs if the range of the inputs exceeds the 0 to 1 range, because when the loss function try to compare the result with the original data, it will always mismatch.

In this case, the best thing you can do is to normalize your inputs so it varies between 0 and 1 just like your outputs.

0
votes

I took a quick look at the dataset you're using. One problem I could find is that your data contains negative values, but your autoencoder has a sigmoid activation function on the last layer. Thus your output is mapped to the range between zero and one. So you'll need to either normalize the data or change the activation function on the last layer to for example tanh, if your values are between -1 and 1.

Also I'm not sure why you should use binary cross-entropy, as the loss function, I think mean squared error (mse) would be more suitable here. According to https://stats.stackexchange.com/a/296277 binary cross-entropy may be asymmetric in some unwanted cases.