使用 Tensorflow 的 Google Colab Bert 实例化错误

Question

我正在尝试在 Colab 上使用 Tensorflow 构建 Bert 模型。这段代码几周前就可以完美运行。现在，如果我尝试实例化模型，则会收到以下错误：

Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFBertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing TFBertModel from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFBertModel from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFBertModel were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-b0e769ef7890> in <cell line: 7>()
      5 SC_mask_layer = Input(shape=(max_seq_length,), dtype=tf.int32, name="attention_mask")
      6 SC_bert_model = TFBertModel.from_pretrained("bert-base-uncased")
----> 7 SC_pooler_output = SC_bert_model(SC_input_layer, attention_mask=SC_mask_layer)[1]  # Estrai il secondo output, che è il pooler_output
      8 
      9 # Aggiungi un layer di Dropout

36 frames
/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/type_spec.py in type_spec_from_value(value)
   1002         3, "Failed to convert %r to tensor: %s" % (type(value).__name__, e))
   1003 
-> 1004   raise TypeError(f"Could not build a TypeSpec for {value} of "
   1005                   f"unsupported type {type(value)}.")
   1006 

TypeError: Exception encountered when calling layer 'embeddings' (type TFBertEmbeddings).

Could not build a TypeSpec for name: "tf.debugging.assert_less_5/assert_less/Assert/Assert"
op: "Assert"
input: "tf.debugging.assert_less_5/assert_less/All"
input: "tf.debugging.assert_less_5/assert_less/Assert/Assert/data_0"
input: "tf.debugging.assert_less_5/assert_less/Assert/Assert/data_1"
input: "tf.debugging.assert_less_5/assert_less/Assert/Assert/data_2"
input: "Placeholder"
input: "tf.debugging.assert_less_5/assert_less/Assert/Assert/data_4"
input: "tf.debugging.assert_less_5/assert_less/y"
attr {
  key: "summarize"
  value {
    i: 3
  }
}
attr {
  key: "T"
  value {
    list {
      type: DT_STRING
      type: DT_STRING
      type: DT_STRING
      type: DT_INT32
      type: DT_STRING
      type: DT_INT32
    }
  }
}
 of unsupported type <class 'tensorflow.python.framework.ops.Operation'>.

Call arguments received by layer 'embeddings' (type TFBertEmbeddings):
  • input_ids=<KerasTensor: shape=(None, 128) dtype=int32 (created by layer 'input_ids')>
  • position_ids=None
  • token_type_ids=<KerasTensor: shape=(None, 128) dtype=int32 (created by layer 'tf.fill_5')>
  • inputs_embeds=None
  • past_key_values_length=0
  • training=False

模型的代码是：

SC_input_layer = Input(shape=(max_seq_length,), dtype=tf.int32, name="input_ids")
SC_mask_layer = Input(shape=(max_seq_length,), dtype=tf.int32, name="attention_mask")
SC_bert_model = TFBertModel.from_pretrained("bert-base-uncased")
SC_pooler_output = SC_bert_model(SC_input_layer, attention_mask=SC_mask_layer)[1]  

# Aggiungi un layer di Dropout
SC_dropout_layer = Dropout(dropout_rate)(SC_pooler_output)
SC_output_layer = Dense(6, activation='sigmoid')(SC_dropout_layer)
SC_model = Model(inputs=[SC_input_layer, SC_mask_layer], outputs=SC_output_layer)

我发现安装tensorflow 2.10.0它可以工作，但是使用Google Colab我的CUDA版本有问题并且使用tensorflow 2.10它无法识别GPU。 该代码几周前就可以工作，有人有解决方案吗？

编辑：同样的错误出现在 Kaggle 上。

Answer 1

更新：该问题与变形金刚的版本有关。使用4.31.0版本应该可以解决。