Neural network: estimating sine wave frequency

Question

With an objective of learning Keras LSTM and RNNs, I thought to create a simple problem to work on: given a sine wave, can we predict its frequency?

I wouldn't expect a simple neural network to be able to predict the frequency, given that the notion of time is important here. However, even with LSTMs, I am unable to learn the frequency; I'm able to learn a trivial zero as the estimated frequency (even for train samples).

Here's the code to create the train set.

import numpy as np
import matplotlib.pyplot as plt

def create_sine(frequency):
    return np.sin(frequency*np.linspace(0, 2*np.pi, 2000))

train_x = np.array([create_sine(x) for x in range(1, 300)])
train_y = list(range(1, 300))

Now, here's a simple neural network for this example.

from keras.models import Model
from keras.layers import Dense, Input, LSTM

input_series = Input(shape=(2000,),name='Input')
dense_1 = Dense(100)(input_series)
pred = Dense(1, activation='relu')(dense_1)
model = Model(input_series, pred)
model.compile('adam','mean_absolute_error')
model.fit(train_x[:100], train_y[:100], epochs=100)

As expected, this NN doesn't learn anything useful. Next, I tried a simple LSTM example.

input_series = Input(shape=(2000,1),name='Input')
lstm = LSTM(100)(input_series)
pred = Dense(1, activation='relu')(lstm)
model = Model(input_series, pred)
model.compile('adam','mean_absolute_error')
model.fit(train_x[:100].reshape(100, 2000, 1), train_y[:100], epochs=100)

However, this LSTM based model also doesn't learn anything useful.

Is it possible that at high x instances that it just looks like a blob? — Jarad
Careful: if the result is zero, you are on the constant part of the relu and there is nothing to be learned in a constant function. — Daniel Möller
Sorry, I misunderstood the question, I thought you wanted to predict the next value. Now I see that you want the frequency. — silgon
@DanielMöller: Thanks. What counter remedy would you suggest? — Nipun Batra

Maxim Maxim · Accepted Answer · 2017-12-27T11:40:21

Why doesn't it learn?

You think it's a simple problem to train an RNN on, but actually your setup isn't easy for the network at all:

As already mentioned, there's lack of important samples. You throw so much data into it (300 * 2000 points), but the actual target (frequency) is seen only once by the network. Even if the network does learn something, there's high chance it will overfit.
Inconsistent data. Remember that RNNs are good at capturing similar patterns in the series data. For instance, in NLP all sentences in the corpus are governed by the same language rules and more sentences help RNN to understand these rules better, i.e., more data helps.

In your case, the series with different frequencies aren't very much alike: compare the sine with frequency=1 and frequency=100. This kind of diversity in the data makes it harder to learn, not easier. It doesn't mean that the frequency is impossible for an RNN to learn, it simply means that you shouldn't be surprised that a trivial RNN like yours has hard time.
Data scale. Changing the frequency from 1 to 300, changes the scale of both x and y by two orders of magnitude, which may be problematic for any neural network.

Solution

Since your goal is rather educational, I solved the second and third items simply by limiting the target frequency to 10, so that scaling and distribution diversity isn't much of an issue (you are welcome to try different values here: you should see that increasing this one parameter to, say, 50 makes the task much more complex).

The first item is solved by giving the RNN 10 examples of each frequency, instead of just one. I've also added one more hidden layer to increase network flexibility, plus a simple regularizer (Dropout layer).

The complete code:

import numpy as np
from keras.models import Model
from keras.layers import Input, Dense, Dropout, LSTM

max_freq = 10
time_steps = 100

def create_sine(frequency, offset):
  return np.sin(frequency * np.linspace(offset, 2 * np.pi + offset, time_steps))

train_y = list(range(1, max_freq)) * 10
train_x = np.array([create_sine(freq, np.random.uniform(0,1)) for freq in train_y])
train_y = np.array(train_y)

input_series = Input(shape=(time_steps, 1), name='Input')
lstm = LSTM(units=100)(input_series)
hidden = Dense(units=100, activation='relu')(lstm)
dropout = Dropout(rate=0.1)(hidden)
output = Dense(units=1, activation='relu')(dropout)

model = Model(input_series, output)
model.compile('adam', 'mean_squared_error')
model.fit(train_x.reshape(-1, time_steps, 1), train_y, epochs=200)

# Trying the network on the same data
test_x = train_x.reshape(-1, time_steps, 1)
test_y = train_y
predicted = model.predict(test_x).reshape([-1])
print()
print((predicted - train_y)[:12])
print(np.mean(np.abs(predicted - train_y)))

The output:

max_freq=10

[-0.05612183 -0.01982236 -0.03744316 -0.02568841 -0.11959982 -0.0770483
  0.04643679  0.12057972 -0.00625324 -0.00724655 -0.16919005 -0.04512954]
0.0503574344847

max_freq=20 (everything else is the same)

[ 0.51365542  0.09269333 -0.009691    0.0619092   0.09852839  0.04378462
  0.01430321 -0.01953268  0.00722599  0.02558327 -0.04520988 -0.0614748 ]
0.146024380232

max_freq=30 (everything else is the same)

[-0.28205156 -0.28922796 -0.00569081 -0.21314907  0.1068716   0.23497915
  0.23975039  0.25955486  0.26333141  0.24235058  0.08320332 -0.03686047]
0.406703719805

Note that results are random and actually increasing the max_freq increases the changes of divergence. But even when it converges, the performance doesn't improve despite having more data, instead gets worse and pretty fast.

Neural network: estimating sine wave frequency

3 Answers

Why doesn't it learn?

Solution