在Keras elmo嵌入层有0个参数？这是正常的吗？

Question

所以我使用GloVe和我的模型一起工作，但现在我改为Elmo（参考GrasHub上的Keras代码Elmo Keras Github，utils.py

但是，当我打印model.summary时，我在ELMo嵌入层中得到0个参数，这与我使用Glove时的情况不同吗？如果不能，请告诉我我做错了什么使用手套我有超过20万个参数

##-------->  When I was using Glove  Embedding Layer
word_embedding_layer = emb.get_keras_embedding(#dropout = emb_dropout,
                                            trainable = True,
                                            input_length = sent_maxlen, 
                                            name='word_embedding_layer') 


## --------> Deep layers
pos_embedding_layer = Embedding(output_dim =pos_tag_embedding_size, #5
                         input_dim = len(SPACY_POS_TAGS),
                         input_length = sent_maxlen, #20
                         name='pos_embedding_layer')
latent_layers = stack_latent_layers(num_of_latent_layers)



##--------> 6] Dropout 
dropout = Dropout(0.1) 

## --------> 7]Prediction
predict_layer = predict_classes()


## --------> 8] Prepare input features, and indicate how to embed them
inputs = [Input((sent_maxlen,), dtype='int32', name='word_inputs'),
            Input((sent_maxlen,), dtype='int32', name='predicate_inputs'),
            Input((sent_maxlen,), dtype='int32', name='postags_inputs')]


## --------> 9] ELMo Embedding and Concat all inputs and run on deep network
from elmo import ELMoEmbedding
import utils 

idx2word = utils.get_idx2word()
ELmoembedding1 = ELMoEmbedding(idx2word=idx2word, output_mode="elmo", trainable=True)(inputs[0]) # These two are interchangeable
ELmoembedding2 = ELMoEmbedding(idx2word=idx2word, output_mode="elmo", trainable=True)(inputs[1]) # These two are interchangeable


embeddings = [ELmoembedding1, 
               ELmoembedding2,
               pos_embedding_layer(inputs[3])]

con1 = keras.layers.concatenate(embeddings)

## --------> 10]Build model 

outputI = predict_layer(dropout(latent_layers(con1)))
model = Model(inputs, outputI)

model.compile(optimizer='adam',
                           loss='categorical_crossentropy',
                           metrics=['categorical_accuracy'])
model.summary()

Trials:

注意：我尝试使用带有Keras代码的TF-Hub Elmo，但输出始终是2D张量[即使我将其更改为'Elmo'设置和'LSTM'而不是默认']因此我无法与POS_embedding_layer连接。我尝试重塑，但最终我得到相同的问题总参数0。

Answer 1

从TF-Hub描述（https://tfhub.dev/google/elmo/2）来看，单个词的嵌入是不可训练的。只有嵌入和LSTM层的加权和。因此，您应该在ELMo级别获得4个可训练参数。

我能够使用StrongIO's example on Github中定义的类获得可训练参数。该示例仅提供一个类，其中输出是默认层，对于每个输入示例（实质上是文档/句子编码器）是1024向量。要访问每个单词（elmo图层）的嵌入，需要进行一些更改，如this issue中所示：

class ElmoEmbeddingLayer(Layer):
    def __init__(self, **kwargs):
        self.dimensions = 1024
        self.trainable=True
        super(ElmoEmbeddingLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        self.elmo = hub.Module('https://tfhub.dev/google/elmo/2', trainable=self.trainable,
                               name="{}_module".format(self.name))

        self.trainable_weights += K.tf.trainable_variables(scope="^{}_module/.*".format(self.name))
        super(ElmoEmbeddingLayer, self).build(input_shape)

    def call(self, x, mask=None):
        result = self.elmo(
            K.squeeze(
                K.cast(x, tf.string), axis=1
            ),
            as_dict=True,
            signature='default',
            )['elmo']
        return result

    def compute_output_shape(self, input_shape):
        return (input_shape[0], None, self.dimensions)

您可以将ElmoEmbeddingLayer与POS层堆叠在一起。 作为一个更一般的例子，可以在一维ConvNet模型中使用ELMo嵌入进行分类：

elmo_input_layer = Input(shape=(None, ), dtype="string")
elmo_output_layer = ElmoEmbeddingLayer()(elmo_input_layer)

conv_layer = Conv1D(
    filters=100,
    kernel_size=3,
    padding='valid',
    activation='relu',
    strides=1)(elmo_output_layer)

pool_layer = GlobalMaxPooling1D()(conv_layer)
dense_layer = Dense(32)(pool_layer)
output_layer = Dense(1, activation='sigmoid')(dense_layer)

model = Model(
    inputs=elmo_input_layer,
    outputs=output_layer)
model.summary()

模型摘要如下所示：

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_62 (InputLayer)        (None, None)              0         
_________________________________________________________________
elmo_embedding_layer_13 (Elm (None, None, 1024)        4         
_________________________________________________________________
conv1d_46 (Conv1D)           (None, None, 100)         307300    
_________________________________________________________________
global_max_pooling1d_42 (Glo (None, 100)               0         
_________________________________________________________________
dense_53 (Dense)             (None, 32)                3232      
_________________________________________________________________
dense_54 (Dense)             (None, 1)                 33        
=================================================================
Total params: 310,569
Trainable params: 310,569
Non-trainable params: 0
_________________________________________________________________