Tensorflow Polynomial Linear Regression curve fit

Question

I have created this Linear regression model using Tensorflow (Keras). However, I am not getting good results and my model is trying to fit the points around a linear line. I believe fitting points around degree 'n' polynomial can give better results. I have looked googled how to change my model to polynomial linear regression using Tensorflow Keras, but could not find a good resource. Any recommendation on how to improve the prediction?

I have a large dataset. Shuffled it first and then spited to 80% training and 20% Testing. Also dataset is normalized.

1) Building model:

def build_model():

  model = keras.Sequential()
  model.add(keras.layers.Dense(units=300, input_dim=32))

  model.add(keras.layers.Activation('sigmoid'))
  model.add(keras.layers.Dense(units=250))

  model.add(keras.layers.Activation('tanh'))
  model.add(keras.layers.Dense(units=200))

  model.add(keras.layers.Activation('tanh'))
  model.add(keras.layers.Dense(units=150))

  model.add(keras.layers.Activation('tanh'))
  model.add(keras.layers.Dense(units=100))

  model.add(keras.layers.Activation('tanh'))
  model.add(keras.layers.Dense(units=50))

  model.add(keras.layers.Activation('linear'))
  model.add(keras.layers.Dense(units=1))  

  #sigmoid tanh softmax relu

  optimizer = tf.train.RMSPropOptimizer(0.001,  
                                        decay=0.9,
                                        momentum=0.0,
                                        epsilon=1e-10,
                                        use_locking=False,
                                        centered=False,
                                        name='RMSProp')

  #optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1)

  model.compile(loss='mse',
                optimizer=optimizer,
                metrics=['mae'])
  return model

model = build_model()
model.summary()

2) Train the model:

class PrintDot(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs):
    if epoch % 100 == 0: print('')
    print('.', end='')

EPOCHS = 500

# Store training stats
history = model.fit(train_data, train_labels, epochs=EPOCHS, 
                    validation_split=0.2, verbose=1,
                    callbacks=[PrintDot()])

3) plot Train loss and val loss enter image description here

4) Stop When results does not get improved enter image description here

5) Evaluate the result

[loss, mae] = model.evaluate(test_data, test_labels, verbose=0)
#Testing set Mean Abs Error: 1.9020842795676374

6) Predict:

test_predictions = model.predict(test_data).flatten()

enter image description here

7) Prediction error: enter image description here

tachyontraveler tachyontraveler · Accepted Answer · 2019-02-25T21:21:30

Polynomial regression is a linear regression with some extra additional input features which are the polynomial functions of original input features. i.e.; let the original input features are : (x₁,x₂,x₃,...)

Generate a set of polynomial functions by adding some transformations of the original features, for example: (x₁², x₂³, x₁³x₂,...). One may decide which all functions are to be included depending on their constraints such as intuition on correlation to the target values, computational resources, and training time.

Append these new features to the original input feature vector. Now the transformed input feature vector has a size of len(x₁,x₂,x₃,...) + len(x₁², x₂³, x₁³x₂,...)

Further, this updated set of input features (x₁,x₂,x₃,x₁², x₂³, x₁³x₂,...) is feeded into the normal linear regression model. ANN's architecture may be tuned again to get the best trained model.

PS: I see that your network is huge while the number of inputs is only 32 - this is not a common scale of architecture. Even in this particular linear model, reducing the hidden layers to one or two hidden layers may help in training better models (It's a suggestion with an assumption that this particular dataset is similar to other generally seen regression datasets)

Tensorflow Polynomial Linear Regression curve fit

2 Answers