使用keras RL搭建agent玩太空侵略者，遇到“AttributeError: 'int' object has no attribute 'shape'”错误

Question

正如标题所说，在按照教程使用 keras RL 制作强化学习代理时，我一直遇到错误。其代码如下：

import gym 
import random
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Convolution2D
from tensorflow.keras.optimizers import Adam
from rl.agents import DQNAgent
from rl.memory import SequentialMemory
from rl.policy import LinearAnnealedPolicy, EpsGreedyQPolicy

def build_model(height, width, channels, actions):
    model = Sequential()
    model.add(Convolution2D(32, (8,8), strides=(4,4), activation='relu', input_shape=(3,height, width, channels)))
    model.add(Convolution2D(64, (4,4), strides=(2,2), activation='relu'))
    model.add(Convolution2D(64, (3,3), activation='relu'))
    model.add(Flatten())
    model.add(Dense(512, activation='relu'))
    model.add(Dense(256, activation='relu'))
    model.add(Dense(actions, activation='linear'))
    return model

def build_agent(model, actions):
    policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.2, nb_steps=10000)
    memory = SequentialMemory(limit=1000, window_length=3)
    dqn = DQNAgent(model=model, memory=memory, policy=policy,
                  enable_dueling_network=True, dueling_type='avg', 
                   nb_actions=actions, nb_steps_warmup=1000
                  )
    return dqn

env = gym.make('SpaceInvaders-v4')
height, width, channels = env.observation_space.shape
actions = env.action_space.n
env.unwrapped.get_action_meanings()


model = build_model(height, width, channels, actions)
model.summary()

dqn = build_agent(model, actions)
dqn.compile(Adam(learning_rate=1e-4))

dqn.fit(env, 10000, visualize=False, verbose=2)

在 fit 代码之前，代码似乎没有问题，它给出了以下错误： AttributeError: 'int' 对象没有属性 'shape'

我相信 keras-RL 本身存储的代码是：

def zeroed_observation(observation):
    """Return an array of zeros with same shape as given observation

    # Argument
        observation (list): List of observation
    
    # Return
        A np.ndarray of zeros with observation.shape
    """
    if hasattr(observation, 'shape'):
        return np.zeros(observation.shape)
    if isinstance(observation, dict):
        keys = observation.keys()
        obs = dict()
        for key in keys:
            obs[key] = np.zeros(observation[key].shape)  <--- This line is the problem
        return obs
    elif hasattr(observation, '__iter__'):
        out = []
        for x in observation:
            out.append(zeroed_observation(x))
        return out
    else:
        return 0.

我不完全确定我做错了什么，因为我正在按照 T 的教程进行操作，但没有任何效果。我正在使用 python 3.10、tensorflow 2.11.0、keras-rl2 1.0.4 和 gym[atari]==0.18.0

我试过修改 dqn.fit 行以包含更多可能缺失的属性，但这似乎不起作用 dqn.fit(env, 10000, 1, None, 1, False, 0, None, 10000, None) <--- The attempted fix, im aware not all the values are the same but still, same error no matter changing them

除此之外，我不确定我做错了什么，因为我是 keras-rl 的新手，而且它大部分是自学的。

完整错误：

Traceback (most recent call last):
  File "C:\Users\Thomas Burns\Desktop\Python File Attempt\test Bank Heist Model.py", line 43, in <module>
    dqn.fit(env, nb_steps = 10000, visualize=False, verbose=0)
  File "C:\Users\Thomas Burns\AppData\Roaming\Python\Python310\site-packages\rl\core.py", line 168, in fit
    action = self.forward(observation)
  File "C:\Users\Thomas Burns\AppData\Roaming\Python\Python310\site-packages\rl\agents\dqn.py", line 223, in forward
    state = self.memory.get_recent_state(observation)
  File "C:\Users\Thomas Burns\AppData\Roaming\Python\Python310\site-packages\rl\memory.py", line 107, in get_recent_state
    state.insert(0, zeroed_observation(state[0]))
  File "C:\Users\Thomas Burns\AppData\Roaming\Python\Python310\site-packages\rl\memory.py", line 63, in zeroed_observation
    out.append(zeroed_observation(x))
  File "C:\Users\Thomas Burns\AppData\Roaming\Python\Python310\site-packages\rl\memory.py", line 58, in zeroed_observation
    obs[key] = np.zeros(observation[key].shape)
AttributeError: 'int' object has no attribute 'shape'

Answer 1

这解决了吗我有同样的问题。