preprocessing big dataset with different size for RNN with tensorflow2.0

3
votes

I have a code to run RNN using Keras. Recently, I increased the size of my samples, but the code crashed during the preprocessing stage due to memory. I think it's because I use brute force to make the windows, meaning I am using for loops and appending all the windows into a list to form the input data. Therefore, I'm trying to use tersorflow.data.Dataset built-in functions. However, my original dataset has 990 samples with different time steps. I can't figure out how to use those built-in functions to make sure the window was not obtained across samples (window should only be sweeping within each sample).

My original samples each have a size ((# of timesteps, 305 features), (# of timesteps, 299 features)). My goal is to have little windows with a size (3,305) and the target for this window has a size (1,299). That is, the window has 3 time steps and the corresponding target is from the third time steps.

Here is an example.

Original data contains (sample and label) pair

sample1=[[1,1],[2,2],[3,3],[4,4],[5,5]]

label1=[[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5]]

sample2=[[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12]]

label2=[[6,6,6],[7,7,7],[8,8,8],[9,9,9],[10,10,10],[11,11,11],[12,12,12]]

sample=[sample1,sample2]

label=[label1,label2]

I would like to get train and target pairs like this

[1,1],[2,2],[3,3] -> [3,3,3]

[2,2],[3,3],[4,4] -> [4,4,4]

[3,3],[4,4],[5,5] -> [5,5,5]

[6,6],[7,7],[8,8] -> [8,8,8]

[7,7],[8,8],[9,9] -> [9,9,9]

[8,8],[9,9],[10,10] -> [10,10,10]

[9,9],[10,10],[11,11] -> [11,11,11]

[10,10],[11,11],[12,12] -> [12,12,12]

Note that if I stack all my samples, there will be about 4950000 number of time steps. My current approach was simply creating an empty list and appending little windows into the list. This ends up with an even bigger array.

Please let me know how I should achieve this goal without memory crashing. Thanks in advance.

Here is my attempt of using tersorflow.data.Dataset built-in function:


dataset = tf.data.Dataset.from_generator(

                   lambda: itertools.zip_longest(sample, label),

                   output_types = (tf.float32, tf.float32),

                   output_shapes = (tf.TensorShape([None,  2]),

                   tf.TensorShape([None, 3])))


dataset_window = dataset.window(3, shift=1, drop_remainder=True)

tf.print(dataset_window)

This gives me

<DatasetV1Adapter shapes: (DatasetSpec(TensorSpec(shape=(None, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(None, 3), dtype=tf.float32, name=None), TensorShape([]))), types: (DatasetSpec(TensorSpec(shape=(None, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(None, 3), dtype=tf.float32, name=None), TensorShape([])))>

I don't know why the TensorShape is empty. I also don't know if doing this will save my code from memory crashing.

1
pythontensorflow2.0
Rather than a print, try to use as_numpy_iterator on your dataset to see what is inside. - Lescurel
Hi, thanks for replying. I tried it on dataset_window. It shows 'DatasetV1Adapter' object has no attribute 'as_numpy_iterator' - cathy
It's not an attribute, it's a method (i.e you're supposed to call as_numpy_iterator()). The documentation that I linked in my previous comment gives a good example on how to use it. - Lescurel
I followed the link you sent. Specifically, I did "for element in dataset_window.as_numpy_iterator():", and it gave me that error. I guess it was because I was not using tf.data.Dataset.from_tensor_slices() like the documentation. I use tf.data.Dataset.from_generator because my dataset has different length. - cathy

1 Answers

0
votes

Even though the TensorShape is empty, it also displays that the elements have shape=(None, 2),shape=(None, 3). You can print the elements to see what is inside.

If you're using Tensorflow 2.0 (or below), window will give you DatasetV1Adapter type and for the later versions window will give you WindowDataset type.

Tensorflow 1.x:

%tensorflow_version 1.x
import tensorflow as tf
import itertools
tf.compat.v1.enable_eager_execution()

sample1=[[1,1],[2,2],[3,3],[4,4],[5,5]]

label1=[[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5]]

sample2=[[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12]]

label2=[[6,6,6],[7,7,7],[8,8,8],[9,9,9],[10,10,10],[11,11,11],[12,12,12]]

sample=[sample1,sample2]

label=[label1,label2]

dataset = tf.data.Dataset.from_generator(

                   lambda: itertools.zip_longest(sample, label),

                   output_types = (tf.float32, tf.float32),

                   output_shapes = (tf.TensorShape([None,  2]),

                   tf.TensorShape([None, 3])))

dataset_window = dataset.window(2, shift=1, drop_remainder=True)

print(dataset_window)

for i in dataset_window:
    for j in i:
      for k in j:
        print(k)

Output:

TensorFlow 1.x selected.
<DatasetV1Adapter shapes: (DatasetSpec(TensorSpec(shape=(?, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(?, 3), dtype=tf.float32, name=None), TensorShape([]))), types: (DatasetSpec(TensorSpec(shape=(?, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(?, 3), dtype=tf.float32, name=None), TensorShape([])))>
tf.Tensor(
[[1. 1.]
 [2. 2.]
 [3. 3.]
 [4. 4.]
 [5. 5.]], shape=(5, 2), dtype=float32)
tf.Tensor(
[[ 6.  6.]
 [ 7.  7.]
 [ 8.  8.]
 [ 9.  9.]
 [10. 10.]
 [11. 11.]
 [12. 12.]], shape=(7, 2), dtype=float32)
tf.Tensor(
[[1. 1. 1.]
 [2. 2. 2.]
 [3. 3. 3.]
 [4. 4. 4.]
 [5. 5. 5.]], shape=(5, 3), dtype=float32)
tf.Tensor(
[[ 6.  6.  6.]
 [ 7.  7.  7.]
 [ 8.  8.  8.]
 [ 9.  9.  9.]
 [10. 10. 10.]
 [11. 11. 11.]
 [12. 12. 12.]], shape=(7, 3), dtype=float32)

Tensorflow 2.x:

%tensorflow_version 2.x
import tensorflow as tf
import itertools

sample1=[[1,1],[2,2],[3,3],[4,4],[5,5]]

label1=[[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5]]

sample2=[[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12]]

label2=[[6,6,6],[7,7,7],[8,8,8],[9,9,9],[10,10,10],[11,11,11],[12,12,12]]

sample=[sample1,sample2]

label=[label1,label2]

dataset = tf.data.Dataset.from_generator(

                   lambda: itertools.zip_longest(sample, label),

                   output_types = (tf.float32, tf.float32),

                   output_shapes = (tf.TensorShape([None,  2]),

                   tf.TensorShape([None, 3])))

dataset_window = dataset.window(2, shift=1, drop_remainder=True)

print(dataset_window)

for i in dataset_window:
    for j in i:
      for k in j:
        print(k)

Output:

<WindowDataset shapes: (DatasetSpec(TensorSpec(shape=(None, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(None, 3), dtype=tf.float32, name=None), TensorShape([]))), types: (DatasetSpec(TensorSpec(shape=(None, 2), dtype=tf.float32, name=None), TensorShape([])), DatasetSpec(TensorSpec(shape=(None, 3), dtype=tf.float32, name=None), TensorShape([])))>
tf.Tensor(
[[1. 1.]
 [2. 2.]
 [3. 3.]
 [4. 4.]
 [5. 5.]], shape=(5, 2), dtype=float32)
tf.Tensor(
[[ 6.  6.]
 [ 7.  7.]
 [ 8.  8.]
 [ 9.  9.]
 [10. 10.]
 [11. 11.]
 [12. 12.]], shape=(7, 2), dtype=float32)
tf.Tensor(
[[1. 1. 1.]
 [2. 2. 2.]
 [3. 3. 3.]
 [4. 4. 4.]
 [5. 5. 5.]], shape=(5, 3), dtype=float32)
tf.Tensor(
[[ 6.  6.  6.]
 [ 7.  7.  7.]
 [ 8.  8.  8.]
 [ 9.  9.  9.]
 [10. 10. 10.]
 [11. 11. 11.]
 [12. 12. 12.]], shape=(7, 3), dtype=float32)