TensorFlow: information/variable remembered between each session.run()?

Question

I am learning TensorFlow from the example at: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb

In the following code, during the training stage, each sess.run() was fed in one data point.

# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
  :
  :
with tf.Session() as sess:
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})
   :

Based on the definition of the optimizer, it is trying to minimize cost. For a cost function J(r), an optimizer should update the parameter r of the cost function using:

r := r - alpha* dJ(r)/dr where alpha is the learning rate

For each data point fed in, it will update the parameter r once, which means the optimizer remembers the results of the previous fed.

Does this mean for the TensorFlow session.run(), it does store the optimizer results from the previous session.run()?

So how would a session() defined? And is everything computed in the same session remembered through each run() until a the session() is over ? Thanks!

chris chris · Accepted Answer · 2016-12-29T01:12:29

When you run the optimizer, it adjusts the parameters to minimize the loss function. When you run it multiple times, it keeps adjusting the parameters over time - none of the parameters get reset.

You can validate this by using sess.run on one of your weight variables after each time you run the optimizer.

TensorFlow: information/variable remembered between each session.run()?

1 Answers