How to slice a tensor with None dimension in Tensorflow

Question

I want to slice a tensor in "None" dimension.

For example,

tensor = tf.placeholder(tf.float32, shape=[None, None, 10], name="seq_holder")
sliced_tensor = tensor[:,1:,:] # it works well!

but

# Assume that tensor's shape will be [3,10, 10]
tensor = tf.placeholder(tf.float32, shape=[None, None, 10], name="seq_holder")
sliced_seq = tf.slice(tensor, [0,1,0],[3, 9, 10]) # it doens't work!

It is same that i get a message when i used another place_holder to feed size parameter for tf.slice().

The second methods gave me "Input size (depth of inputs) must be accessible via shape inference" error message.

I'd like to know what's different between two methods and what is more tensorflow-ish way.

[Edited] Whole code is below

import tensorflow as tf
import numpy as np

print("Tensorflow for tests!")

vec_dim = 5
num_hidden = 10
# method 1
input_seq1 = np.random.random([3,7,vec_dim])

# method 2
input_seq2 = np.random.random([5,10,vec_dim])
shape_seq2 = [5,9,vec_dim]
# seq: [batch, seq_len]
seq = tf.placeholder(tf.float32, shape=[None, None, vec_dim], name="seq_holder")

# Method 1
sliced_seq = seq[:,1:,:]

# Method 2
seq_shape = tf.placeholder(tf.int32, shape=[3])
sliced_seq = tf.slice(seq,[0,0,0], seq_shape)

cell = tf.contrib.rnn.GRUCell(num_units=num_hidden)
init_state = cell.zero_state(tf.shape(seq)[0], tf.float32)

outputs, last_state = tf.nn.dynamic_rnn(cell, sliced_seq, initial_state=init_state)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    # method 1
    # states = sess.run([sliced_seq], feed_dict={seq:input_seq1})
    # print(states[0].shape)

    # method 2
    states = sess.run([sliced_seq], feed_dict={seq:input_seq2, seq_shape:shape_seq2})
    print(states[0].shape)

Do you get the error when you define the operations (when running those two lines) or when you try to execute them (calling .run in a session, for example) with some input? I am having no trouble running those two instructions. — jdehesa
I got an error when i put the sliced tensor(2nd method) to dynamic rnn function. Is it a problem of dynamic_rnn? — Hanjun koo
Hmm, hard to tell... can you maybe edit your question and add more details, maybe a full minimal reproducible example where your issue appears? — jdehesa
I edited my question. There are 2 types of slicing now. Thanks :) — Hanjun koo

jdehesa jdehesa · Accepted Answer · 2017-09-15T13:33:09

Your problem is exactly described by issue #4590

The problem is that tf.nn.dynamic_rnn needs to know the size of the last dimension in the input (the "depth"). Unfortunately, as the issue points out, currently tf.slice cannot infer any output size if any of the slice ranges are not fully known at graph construction time; therefore, sliced_seq ends up having a shape (?, ?, ?).

In your case, the first issue is that you are using a placeholder of three elements to determine the size of the slice; this is not the best approach, since the last dimension should never change (even if you later pass vec_dim, it could cause errors). The easiest solution would be to turn seq_shape into a placeholder of size 2 (or even two separate placeholders), and then do the slicing like:

sliced_seq = seq[:seq_shape[0], :seq_shape[1], :]

For some reason, the NumPy-style indexing seems to have better shape inference capabilities, and this will preserve the size of the last dimension in sliced_seq.

How to slice a tensor with None dimension in Tensorflow

1 Answers