error when training single level LSTM in tensorflow

Question

So I have been trying to train a single layered encoder-decoder network in tensorflow, it is just simply so frustrating given the document is so sparse on explanation, and I have only taken Stanford's CS231n on tensorflow.

So here's the straightforward model:

def simple_model(X,Y, is_training):
    """
    a simple, single layered encoder decoder network, 
    that encodes X of shape (batch_size, window_len, 
    n_comp+1), then decodes Y of shape (batch_size, 
    pred_len+1, n_comp+1), of which the vector Y[:,0,
    :], is simply [0,...,0,1] * batch_size, so that 
    it starts the decoding
    """

    num_units = 128
    window_len = X.shape[1]
    n_comp = X.shape[2]-1
    pred_len = Y.shape[1]-1

    init = tf.contrib.layers.variance_scaling_initializer()
    encoder_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units)
    encoder_output, encoder_state = tf.nn.dynamic_rnn(
                         encoder_cell,X,dtype = tf.float32)
    decoder_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units)
    decoder_output, _ = tf.nn.dynamic_rnn(decoder_cell,
                                         encoder_output,
                             initial_state = encoder_state)
    # we expect the shape to be of the shape of Y
    print(decoder_output.shape)
    proj_layer = tf.layers.dense(decoder_output, n_comp)
    return proj_layer

now I try to set up the training details:

tf.reset_default_graph()
X = tf.placeholder(tf.float32, [None, 15, 74])
y = tf.placeholder(tf.float32, [None, 4, 74])
is_training = tf.placeholder(tf.bool)
y_out = simple_model(X,y,is_training)

mean_loss = 0.5*tf.reduce_mean((y_out-y[:,1:,:-1])**2)
optimizer = tf.train.AdamOptimizer(learning_rate=5e-4)

extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
    train_step = optimizer.minimize(mean_loss)

okay then now I get this stupid error

ValueError: Variable rnn/basic_lstm_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

Maybe you are using(defining) rnn/basic_lstm_cell/kernel at multiple places in the same program. So when tensorflow tries to build the graph it fail. Please post more information (complete error message) — walkerlala

Y. Luo Y. Luo · Accepted Answer · 2018-05-03T22:10:16

I'm not sure if I understand this correctly. You have two BasicLSTMCells in your graph. According to the documentation, you probably should use MultiRNNCell like this:

encoder_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units)
decoder_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units)
rnn_layers = [encoder_cell, decoder_cell]
multi_rnn_cell = tf.nn.rnn_cell.MultiRNNCell(rnn_layers)
decoder_output, state = tf.nn.dynamic_rnn(cell=multi_rnn_cell,
                                          inputs=X,
                                          dtype=tf.float32)

If this is not the correct architecture you'd like to have and you need to use the two BasicLSTMCells separately, I think pass different/unique names when defining encoder_cell and decoder_cell will help solve this error. tf.nn.dynamic_rnn will put the cell under a 'rnn' scope. If you don't define the cell name explicitly, that will cause a reuse confusion.

error when training single level LSTM in tensorflow

1 Answers