Teach LSTMs concept of different frequencies

Question

Training an LSTM on a sine wave is simple enough. The LSTM easily understands how to predict the wave hundreds of timesteps into the future.

However, the LSTM becomes grossly incompetent once I try to have it predict a sine wave of a different frequency. Even if I try to train it on many waves of different frequencies, it forgets the first wave to learn the next one. An example of how my LSTM fails when I changed the frequency of the test data:

How do I train my LSTM to recognize the concept of frequency and work on any sinusoid?

Edit:

The model that I am using:

inputs = Input(shape=(self.timesteps, self.features))

bd_seq = Bidirectional(LSTM(128, return_sequences=True,
                                    kernel_regularizer='l2'),
                               merge_mode='sum')(inputs)
bd_sin = Bidirectional(LSTM(32, return_sequences=True,
                                    kernel_regularizer='l2'),
                               merge_mode='sum')(bd_seq)

bd_1 = Bidirectional(LSTM(self.features, activation='linear'),
                             merge_mode='sum')(bd_seq)
bd_2 = Bidirectional(LSTM(self.features, activation='tanh'),
                             merge_mode='sum')(bd_sin)
output = Add()([bd_1, bd_2])

self.model = Model(inputs=inputs, outputs=output)

Neat problem! Have you tried a multi-layer LSTM? That is, the output of one LSTM RNN feeds into the input to another LSTM (the second layer), and so on for say 3 or 4 layers until the final layer outputs to the loss function. This is a common architecture that improves accuracy and might make a difference in a case like this. — David Parks
I am using multilayer with bidirectional LSTMs. The full model is above — Primusa

Daniel Möller Daniel Möller · Accepted Answer · 2018-04-21T19:21:38

"...it forgets the first wave to learn the next one..."

This makes me think... are you training one sequence, then another one, then another??

That will fail, naturally, for any kind of problem with any model.

You must train lots of sequences in the same batch, or, if one sequence at a time, never more than once per epoch.

freqs = list_of_frequencies
sinusoids = []
for freq in freqs:
    sinusoids.append(create_a_sinusoid(freq))

training_data = np.array(sinusoids).reshape((freqs,timesteps,features))

Possible tricks to help the model:

Add the frequency as a feature (for all steps) in the input data (if you know it as input)
Make the model output the frequency (if you know it as output) and train it on frequencies.
You may combine a model that identifies frequences with a model that will read these frequencies for predicting the desired outputs

Teach LSTMs concept of different frequencies

1 Answers