用于太空入侵者的LSTM网络RL（Keras）

Question

我是强化学习的新手，并正在尝试使用LSTM进行太空入侵者特工的强化学习。我尝试使用在此paper中找到的网络，但是我一直遇到麻烦：

-如果我使用conv2D，则LSTM的尺寸不合适，并且出现此错误：

ValueError：输入0与层conv_lst_m2d_1不兼容：预期的ndim = 5，找到的ndim = 4

这是代码：

    self.model = Sequential()
    self.model.add(Conv2D(32,kernel_size=8,strides=4,activation='relu',input_shape=(None,84,84,1)))
    self.model.add(Conv2D(64,kernel_size=4,strides=2,activation='relu'))
    self.model.add(Conv2D(64,kernel_size=3, strides=1,activation='relu'))
    self.model.add(ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))
    self.model.add(Dense(4, activation='relu'))
    self.model.compile(loss='mse', optimizer=Adam(lr=0.0001))
    self.model.summary()

-如果我使用输出5D张量的Conv3D，则不能使用一个图像作为输入：

ValueError：检查输入时出错：预期conv3d_1_input具有5个维度，但数组的形状为（1、84、84、1）]

代码：

    self.model.add(Conv3D(32,kernel_size=8,strides=4,activation='relu',input_shape=(None,84,84,1)))
    self.model.add(Conv3D(64,kernel_size=4,strides=2,activation='relu'))
    self.model.add(Conv3D(64,kernel_size=3, strides=1,activation='relu'))
    self.model.add(ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))
    self.model.add(Dense(4, activation='relu'))
    self.model.compile(loss='mse', optimizer=Adam(lr=0.0001))
    self.model.summary()

（编辑）

[第二网络的网络摘要：

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv3d_1 (Conv3D)            (None, None, 20, 20, 32)  16416     
_________________________________________________________________
conv3d_2 (Conv3D)            (None, None, 9, 9, 64)    131136    
_________________________________________________________________
conv3d_3 (Conv3D)            (None, None, 7, 7, 64)    110656    
_________________________________________________________________
conv_lst_m2d_1 (ConvLSTM2D)  (None, 7, 7, 512)         10618880  
_________________________________________________________________
dense_1 (Dense)              (None, 7, 7, 4)           2052      
=================================================================

并且数据输入形状为：(84, 84, 1)

Answer 1

您需要使用TimeDistributed Conv2D，它告诉您的网络将数据理解为临时数据（我想这是您想要的），并将与LSTM内部形状匹配。

import tensorflow as tf

model = tf.keras.Sequential()

model.add(tf.keras.layers.Input(shape=(None,84,84,1)))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(32,kernel_size=8,strides=4,activation='relu')))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(64,kernel_size=4,strides=2,activation='relu')))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv2D(64,kernel_size=3, strides=1,activation='relu')))

model.add(tf.keras.layers.ConvLSTM2D(512, kernel_size=(3,3), padding='same', return_sequences=False))

model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(4, activation='relu')))

model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(lr=0.0001))

model.summary()

编译返回：

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
time_distributed_12 (TimeDis (None, None, 20, 20, 32)  2080      
_________________________________________________________________
time_distributed_13 (TimeDis (None, None, 9, 9, 64)    32832     
_________________________________________________________________
time_distributed_14 (TimeDis (None, None, 7, 7, 64)    36928     
_________________________________________________________________
conv_lst_m2d_3 (ConvLSTM2D)  (None, 7, 7, 512)         10618880  
_________________________________________________________________
time_distributed_15 (TimeDis (None, 7, 7, 4)           2052      
=================================================================
Total params: 10,692,772
Trainable params: 10,692,772
Non-trainable params: 0
_________________________________________________________________

Answer 2

首先尝试打印模型的输入和输出详细信息：-

o / p将是这样-

标记模型的输入：

[{'name': 'input', 'index': 451, 'shape': array([  1, 160, 160,   3],
dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0,
0)}]

标记输出模型：

[{'name': 'embeddings', 'index': 450, 'shape': array([  1, 512], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0)}]

一旦获得详细信息，根据详细信息，您将必须提供input_shape的值。