我将迷你批量数据提供给模型,我只想知道如何处理损失。我可以累积损失,然后调用后退,如:
...
def neg_log_likelihood(self, sentences, tags, length):
self.batch_size = sentences.size(0)
logits = self.__get_lstm_features(sentences, length)
real_path_score = torch.zeros(1)
total_score = torch.zeros(1)
if USE_GPU:
real_path_score = real_path_score.cuda()
total_score = total_score.cuda()
for logit, tag, leng in zip(logits, tags, length):
logit = logit[:leng]
tag = tag[:leng]
real_path_score += self.real_path_score(logit, tag)
total_score += self.total_score(logit, tag)
return total_score - real_path_score
...
loss = model.neg_log_likelihood(sentences, tags, length)
loss.backward()
optimizer.step()
我想知道如果积累可能导致梯度爆炸?
那么,我应该调用后向循环:
for sentence, tag , leng in zip(sentences, tags, length):
loss = model.neg_log_likelihood(sentence, tag, leng)
loss.backward()
optimizer.step()
或者,使用平均损失,就像tensorflow中的reduce_mean一样
loss = reduce_mean(losses)
loss.backward()
loss
必须使用迷你批量大小减少mean
。如果你看一下像CrossEntropyLoss这样的原生PyTorch损失函数,就会有一个单独的参数reduction
,默认行为是在迷你批量大小上做mean
。