LSTM network for space-invaders RL (Keras)

0
votes

i´m new to reinforcement learning, and was trying to use LSTM for reinforcement learning for a space invaders agent. I tried to use the network found in this paper but I kept having trouble:

-If i use conv2D the dimensions with the LSTM dont fit and i get this error:

ValueError: Input 0 is incompatible with layer conv_lst_m2d_1: expected ndim=5, found ndim=4

This is the code:

    self.model = Sequential()
    self.model.add(Conv2D(32,kernel_size=8,strides=4,activation='relu',input_shape=(None,84,84,1)))
    self.model.add(Conv2D(64,kernel_size=4,strides=2,activation='relu'))
    self.model.add(Conv2D(64,kernel_size=3, strides=1,activation='relu'))
    self.model.add(ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))
    self.model.add(Dense(4, activation='relu'))
    self.model.compile(loss='mse', optimizer=Adam(lr=0.0001))
    self.model.summary()

-And if I use Conv3D that outputs a 5D tensor I cant use one image as an input:

ValueError: Error when checking input: expected conv3d_1_input to have 5 dimensions, but got array with shape (1, 84, 84, 1)

Code: 

    self.model.add(Conv3D(32,kernel_size=8,strides=4,activation='relu',input_shape=(None,84,84,1)))
    self.model.add(Conv3D(64,kernel_size=4,strides=2,activation='relu'))
    self.model.add(Conv3D(64,kernel_size=3, strides=1,activation='relu'))
    self.model.add(ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))
    self.model.add(Dense(4, activation='relu'))
    self.model.compile(loss='mse', optimizer=Adam(lr=0.0001))
    self.model.summary()

(edit)

Network summary(of the second network): 

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv3d_1 (Conv3D)            (None, None, 20, 20, 32)  16416     
_________________________________________________________________
conv3d_2 (Conv3D)            (None, None, 9, 9, 64)    131136    
_________________________________________________________________
conv3d_3 (Conv3D)            (None, None, 7, 7, 64)    110656    
_________________________________________________________________
conv_lst_m2d_1 (ConvLSTM2D)  (None, 7, 7, 512)         10618880  
_________________________________________________________________
dense_1 (Dense)              (None, 7, 7, 4)           2052      
=================================================================

And data input shape is: (84, 84, 1)

2
pythonmachine-learningkeraslstmreinforcement-learning
The first error you get refers to the ConvLSTM2D layer, therefore your initial choice of layers seems to work. Can you print the network summary and include in the question as well as an input sample/shape? - nickthefreak
The first network doesn't print a summary, because the network isn't even constructing... Just realizing the other one does construct the network and outputs the error when we input the images. - JaimeRomero
The second network is compiling but the input shape of the Conv_3D layer is not working for the shape of your data. On the other hand, the first network is not compiling as the output size of the last Conv_2d layer is not working with the input size of the ConvLSTM layer. - nickthefreak

2 Answers

0
votes

First try to print input and output details of your model:-

o/p will be like this -

Input to marks model: 

[{'name': 'input', 'index': 451, 'shape': array([  1, 160, 160,   3],
dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0,
0)}]

Output of marks model: 

[{'name': 'embeddings', 'index': 450, 'shape': array([  1, 512], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0)}]

once you will get details, according to the details you will have to give the value of input_shape.

0
votes

You need to use TimeDistributed Conv2D, it tells your network to understand the data as temporal (which I guess is what you want) and will match the LSTM inner shape.

import tensorflow as tf

model = tf.keras.Sequential()

model.add(tf.keras.layers.Input(shape=(None,84,84,1)))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(32,kernel_size=8,strides=4,activation='relu')))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(64,kernel_size=4,strides=2,activation='relu')))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(64,kernel_size=3, strides=1,activation='relu')))

model.add(tf.keras.layers.ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(4, activation='relu')))

model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(lr=0.0001))

model.summary()

Compile returns : 

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
time_distributed_12 (TimeDis (None, None, 20, 20, 32)  2080      
_________________________________________________________________
time_distributed_13 (TimeDis (None, None, 9, 9, 64)    32832     
_________________________________________________________________
time_distributed_14 (TimeDis (None, None, 7, 7, 64)    36928     
_________________________________________________________________
conv_lst_m2d_3 (ConvLSTM2D)  (None, 7, 7, 512)         10618880  
_________________________________________________________________
time_distributed_15 (TimeDis (None, 7, 7, 4)           2052      
=================================================================
Total params: 10,692,772
Trainable params: 10,692,772
Non-trainable params: 0
_________________________________________________________________