RNN-LSTM accuracy not improving when training

Question

I am trying to use a keras RNN-LSTM algorithm to classify 9 different sitting positions by using sensor data (X/Y/Z-accelerometer data, X/Y/Z-gyroscope data, euler-angles etc.).

The training data are recorded by placing 3 sensors on 8 test subjects' backs, and making them sit in the 9 different sitting positions. The 3 first test subjects sat for 2 minutes in each position and the last 5 sat for 3 minutes in each position.

The different test-subjects recorded these amounts of data (in rows):

Subject 1: 52824
Subject 2: 51490
Subject 3: 52019
Subject 4: 80613
Subject 5: 70143
Subject 6: 79231
Subject 7: 16027
Subject 8: 15780

All the recordings are recorded as CSV files, with timestamps (the first column in each CSV file) starting at 0,0000 up to the duration of the trial.

The transitions between the phases (where the test subjects are changing sitting positions) are cut out from the dataset, and therefore we have some small "holes" in our time-series.

Combining all these rows together, our X_train set consists of 418127 rows and 39 columns.

np.shape(x_train) // (418127, 39)

And my y_train set is a one-dimensional array containing the correct label corresponding to the row in the x_train set with the same index.

np.shape(y_train) // (418127,)

As a RNN-LSTM model requires a 3d-array as input, I made a method for creating a 3d-array from my X-train-set.


def create_3d_array(array, num_timestamps):

    arr_3d = []
    temp_2d = []
    for i in range(1,len(array)):
        temp_2d.append(array[0])
        if i % NUM_TIMESTAMPS == 0:
            arr_3d.append(temp_2d)
            temp_2d = []
    print("x_train: ", np.shape(arr_3d))
    
    return arr_3d

The shape of the resulting 3d-array is dependent on how long the sequence (# timestamps) is supposed to be.

Question 1: in this particular example, what is a fitting sequence-length for the 3d-array to be given as input to the RNN-LSTM model? Is the model supposed to have long sequences, or can a RNN-LSTM model work just as well on small sequences?

I also one-hot-encoded y_train in order to be able to do multiclass classification.

np.shape(x_train_3d) //(20906,20,39)
np.shape(y_train_3d) //(20906, 9)

I have previously tried with a sequence-length of 20, (20 timestamps for each sample) and this model. I want to keep it as simple as possible, and therefore implement the model as a RNN-LSTM Vanilla.

model = Sequential()
model.add(layers.LSTM(2, activation='relu', input_shape=[x_train.shape[1], x_train.shape[2]]))
model.add(Dense(9, activation='softmax'))
model.compile(optimizer=OPTIM, loss=tf.keras.losses.CategoricalCrossentropy(), metrics=['accuracy'])

model.fit(x=x_train, batch_size=5, y=y_train, epochs=10, shuffle=False)



Epoch 1/10
4182/4182 [==============================] - 28s 7ms/step - loss: 4.9613 - accuracy: 0.7435
Epoch 2/10
4182/4182 [==============================] - 19s 5ms/step - loss: 4.7745 - accuracy: 0.7447
Epoch 3/10
4182/4182 [==============================] - 52s 12ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 4/10
4182/4182 [==============================] - 47s 11ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 5/10
4182/4182 [==============================] - 61s 15ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 6/10
4182/4182 [==============================] - 69s 16ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 7/10
4182/4182 [==============================] - 70s 17ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 8/10
4182/4182 [==============================] - 62s 15ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 9/10
4182/4182 [==============================] - 52s 12ms/step - loss: 4.7731 - accuracy: 0.7447
Epoch 10/10
4182/4182 [==============================] - 58s 14ms/step - loss: 4.7731 - accuracy: 0.7447

The test-data are produced the same way, but with less samples, and with each position recorded for about 20 seconds.

np.shape(x_test) // (8367, 39) 
np.shape(y_test) // (8367,)

The same methods are used on the test-data also, resulting in these shapes:


np.shape(x_test) //(418, 20, 39)
np.shape(y_test) //(418, 9)

Model evaluation:

model.evaluate(x_test,y_test, batch_size=5)

418/418 [==============================] - 2s 6ms/step - loss: 14.3449 - accuracy: 0.1172
[14.344941139221191, 0.11722487956285477]

Question 2: why is the accuracy not improving, and why the evaluation gives such bad results?

Nameless Nameless · Accepted Answer · 2021-03-06T12:20:36

I try to give you some advice since in the past I also found myself with a model that did not improve. First of all, if you see that the model at the training level remains stationary and then at the test level it does not give decent results, you must immediately think about overfitting. Probably since you use a Dense (9) I would try to increase the number and size of the LSTM layer, perhaps adding another one with more cells. After that I see that you are not setting the learning rate, why don't you try a scheduled learning rate? it has given me great results. It also uses batches to train in order to avoid overfit.

RNN-LSTM accuracy not improving when training

1 Answers