Keras 模型突然开始输出张量。如何恢复？

Question

所以我正在学习 DQN 试图解决 Cart Pole env：

import gymnasium as gym
import numpy as np
from rl.agents import DQNAgent
from rl.memory import SequentialMemory
from rl.policy import BoltzmannQPolicy
from tensorflow.python.keras.layers import InputLayer, Dense
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.optimizer_v2.adam import Adam

if __name__ == '__main__':
    env = gym.make("CartPole-v1")

    # tensorflow.compat.v1.experimental.output_all_intermediates(True)

    model = Sequential()
    model.add(InputLayer(input_shape=(1, 4)))
    model.add(Dense(24, activation="relu"))
    # model.add(GRU(24))
    model.add(Dense(24, activation="relu"))
    model.add(Dense(env.action_space.n, activation="linear"))
    model.build()

    print(model.summary())

    agent = DQNAgent(
        model=model,
        memory=SequentialMemory(limit=50000, window_length=1),
        policy=BoltzmannQPolicy(),
        nb_actions=env.action_space.n,
        nb_steps_warmup=100,
        target_model_update=0.01
    )

    agent.compile(Adam(learning_rate=0.001), metrics=["mae"])
    agent.fit(env, nb_steps=100000, visualize=False, verbose=1)

    results = agent.test(env, nb_episodes=10, visualize=True)
    print(np.mean(results.history["episode_reward"]))

    env.close()

一切都很好，我能够解决这个环境，但在某些时候我想尝试添加 GRU 层，看看这将如何影响学习。然后为了让它工作，我使用了

tensorflow.compat.v1.experimental.output_all_intermediates(True)

。现在，即使没有 GRU 层，我也会收到以下错误：

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 1, 24)             120
_________________________________________________________________
dense_1 (Dense)              (None, 1, 24)             600
_________________________________________________________________
dense_2 (Dense)              (None, 1, 2)              50
=================================================================
Total params: 770
Trainable params: 770
Non-trainable params: 0
_________________________________________________________________
None
Traceback (most recent call last):
  File "cart_pole.py", line 25, in <module>
    agent = DQNAgent(
  File "lib\site-packages\rl\agents\dqn.py", line 107, in __init__
    raise ValueError(f'Model output "{model.output}" has invalid shape. DQN expects a model that has one dimension for each action, in this case {self.nb_actions}.')
ValueError: Model output "Tensor("dense_2/BiasAdd:0", shape=(None, 1, 2), dtype=float32)" has invalid shape. DQN expects a model that has one dimension for each action, in this case 2.

我假设正在发生的是添加

tensorflow.compat.v1.experimental.output_all_intermediates(True)

使我的模型输出张量而不是之前输出的内容。将

False

或

None

传递给

output_all_intermediates

根本没有任何效果。如何恢复我的模型以再次与 DQN 代理一起使用？

Answer 1

我认为问题与 model.add(InputLayer(input_shape=(1, 4))) 相关，尝试 input_shape=(4,)。