我已经训练了tensforflow 2.0 keras模型来进行一些自然语言处理。
我基本上在做的是获得不同新闻的标题并预测它们所属的类别。为此,我必须标记这些句子,然后添加0以填充数组以具有与我定义的长度相同的长度:
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
max_words = 1500
tokenizer = Tokenizer(num_words=max_words )
tokenizer.fit_on_texts(x.values)
X = tokenizer.texts_to_sequences(x.values)
X = pad_sequences(X, maxlen = 32)
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Embedding, LSTM, GRU,InputLayer
numero_clases = 5
modelo_sentimiento = Sequential()
modelo_sentimiento.add(InputLayer(input_tensor=tokenizer.texts_to_sequences, input_shape=(None, 32)))
modelo_sentimiento.add(Embedding(max_palabras, 128, input_length=X.shape[1]))
modelo_sentimiento.add(LSTM(256, dropout=0.2, recurrent_dropout=0.2, return_sequences=True))
modelo_sentimiento.add(LSTM(256, dropout=0.2, recurrent_dropout=0.2))
modelo_sentimiento.add(Dense(numero_clases, activation='softmax'))
modelo_sentimiento.compile(loss = 'categorical_crossentropy', optimizer='adam',
metrics=['acc',f1_m,precision_m, recall_m])
print(modelo_sentimiento.summary())
现在受过训练,我想将其例如部署在tensorflow服务中,但是我不知道如何将该预处理(令牌生成器)保存到服务器中,就像制作scikit-learn管道一样,是否可以在此处进行呢?还是我必须保存令牌生成器并自行进行预处理,然后调用经过训练的模型进行预测?
with open('tokenizer_data.pkl', 'wb') as handle:
pickle.dump(
{'tokenizer': tokenizer, 'num_words':num_words, 'maxlen':pad_len}, handle)
然后在要使用它时加载它,
with open("tokenizer_data.pkl", 'rb') as f:
data = pickle.load(f)
tokenizer = data['tokenizer']
num_words = data['num_words']
maxlen = data['maxlen']