我正在尝试实现一个结合了LSTM和CNN模型的预测算法,从 本文. 本质上,该论文提出了一个具有三个分支的模型:一个CNN分支,一个LSTM分支,以及一个结合两者的合并分支。前两个分支只在训练过程中出现,以防止过拟合,并确保最终模型同时针对CNN和LSTM特征进行训练。这是论文中的图(总损失函数中的α、β和gamma只是这些特定损失的权重)。据我了解,这些类似于ResNet和Inception模型中的辅助分支,以确保每一层都对模型输出有贡献。我据此实现了这一点。
def construct_lstm_cnn(look_forward, look_back=30):
cnn = construct_cnn(look_forward, fc=False)
cnn_flatten = Flatten()(cnn.output)
lstm = construct_lstm(look_forward, look_back, 2, fc=False)
#Merged layer (the main branch that will be making prediction after training)
cnn_lstm = concatenate([cnn_flatten, lstm.output])
fc_merged = Dense(500, activation='relu')(cnn_lstm)
drop_merged = Dropout(0.5)(fc_merged)
fc2_merged = Dense(100, activation='relu')(drop_merged)
drop2_merged = Dropout(0.5)(fc2_merged)
fc3_merged = Dense(25 , activation='relu')(drop2_merged)
drop3_merged = Dropout(0.5)(fc3_merged)
pred_merged = Dense(look_forward, activation='linear')(drop3_merged)
#Auxiliary branch for cnn (want to remove at inference time)
fc_cnn = Dense(500, activation='relu')(cnn_flatten)
drop_cnn = Dropout(0.5)(fc_cnn)
fc2_cnn = Dense(100, activation='relu')(drop_cnn)
drop2_cnn = Dropout(0.5)(fc2_cnn)
fc3_cnn = Dense(25 , activation='relu')(drop2_cnn)
drop3_cnn = Dropout(0.5)(fc3_cnn)
pred_cnn_aux = Dense(look_forward, activation='linear')(drop3_cnn)
#Auxiliary branch for lstm (want to remove at inference time)
fc_lstm = Dense(500, activation='relu')(lstm.output)
drop_lstm = Dropout(0.5)(fc_lstm)
fc2_lstm = Dense(100, activation='relu')(drop_lstm)
drop2_lstm = Dropout(0.5)(fc2_lstm)
fc3_lstm = Dense(25 , activation='relu')(drop2_lstm)
drop3_lstm = Dropout(0.5)(fc3_lstm)
pred_lstm_aux = Dense(look_forward, activation='linear')(drop3_lstm)
#Final model with three branches
model = Model(inputs=[cnn.input, lstm.input], outputs=[pred_merged, pred_cnn_aux, pred_lstm_aux], name="lstm-cnn")
return model
然而,我似乎在Keras中找不到一种方法来移除列出的辅助分支。有什么方法可以让我删除那些在推理时没用的层吗?
我为你提供一个简化的例子
这里有完整的模型和所有的分支... 这是要适合的模型。
def construct_lstm_cnn():
inp_lstm = Input((20,30))
lstm = LSTM(32, activation='relu')(inp_lstm)
inp_cnn = Input((32,32,3))
cnn = Conv2D(32, 3, activation='relu')(inp_cnn)
cnn = Flatten()(cnn)
cnn_lstm = Concatenate()([cnn, lstm])
cnn_lstm = Dense(1)(cnn_lstm)
fc_cnn = Dense(32, activation='relu')(cnn)
fc_cnn = Dropout(0.5)(fc_cnn)
fc_cnn = Dense(1)(fc_cnn)
fc_lstm = Dense(32, activation='relu')(lstm)
fc_lstm = Dropout(0.5)(fc_lstm)
fc_lstm = Dense(1)(fc_lstm)
model = Model(inputs=[inp_cnn, inp_lstm], outputs=[cnn_lstm, fc_cnn, fc_lstm])
return model
lstm_cnn = construct_lstm_cnn()
lstm_cnn.compile(...)
lstm_cnn.summary()
lstm_cnn.fit(...)
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_10 (InputLayer) [(None, 32, 32, 3)] 0
__________________________________________________________________________________________________
conv2d_18 (Conv2D) (None, 30, 30, 32) 896 input_10[0][0]
__________________________________________________________________________________________________
input_9 (InputLayer) [(None, 20, 30)] 0
__________________________________________________________________________________________________
flatten_3 (Flatten) (None, 28800) 0 conv2d_18[0][0]
__________________________________________________________________________________________________
lstm_5 (LSTM) (None, 32) 8064 input_9[0][0]
__________________________________________________________________________________________________
dense_13 (Dense) (None, 32) 921632 flatten_3[0][0]
__________________________________________________________________________________________________
dense_15 (Dense) (None, 32) 1056 lstm_5[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 28832) 0 flatten_3[0][0]
lstm_5[0][0]
__________________________________________________________________________________________________
dropout_3 (Dropout) (None, 32) 0 dense_13[0][0]
__________________________________________________________________________________________________
dropout_4 (Dropout) (None, 32) 0 dense_15[0][0]
__________________________________________________________________________________________________
dense_12 (Dense) (None, 1) 28833 concatenate_1[0][0]
__________________________________________________________________________________________________
dense_14 (Dense) (None, 1) 33 dropout_3[0][0]
__________________________________________________________________________________________________
dense_16 (Dense) (None, 1) 33 dropout_4[0][0]
==================================================================================================
为推理时间,在训练后,我们可以简单地用这种方式去除无用的分支。
lstm_cnn_inference = Model(lstm_cnn.input, lstm_cnn.output[0])
lstm_cnn_inference.summary()
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_10 (InputLayer) [(None, 32, 32, 3)] 0
__________________________________________________________________________________________________
conv2d_18 (Conv2D) (None, 30, 30, 32) 896 input_10[0][0]
__________________________________________________________________________________________________
input_9 (InputLayer) [(None, 20, 30)] 0
__________________________________________________________________________________________________
flatten_3 (Flatten) (None, 28800) 0 conv2d_18[0][0]
__________________________________________________________________________________________________
lstm_5 (LSTM) (None, 32) 8064 input_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 28832) 0 flatten_3[0][0]
lstm_5[0][0]
__________________________________________________________________________________________________
dense_12 (Dense) (None, 1) 28833 concatenate_1[0][0]
==================================================================================================
这样一来,我们就只保留了中心分支