如何查找单词列表之间的语义相似性？

Question

输入：

listToStr = 'degeneration agents alpha alternative amd analysis angiogenesis anti anti vegf appears associated based best bevacizumab blindness blood'

我正在使用的代码：

simi = []
tokens = nlp(listToStr) 
length = len(tokens)

for i in range(length):
    #print(i)
    sim = tokens[i].similarity(tokens[i+1])
    simi.append(sim)
print(simi)

错误：

[E040] Attempt to access token at 17, max length 17.

如何删除此错误？

我正在使用spacy。这是它的链接：https://www.geeksforgeeks.org/python-word-similarity-using-spacy/#:~:text=Python%20%7C%20Word%20Similarity%20using%20spaCy,simple%20method%20for%20this%20task.

Answer 1

在for循环内部，由于tokens[i + 1]操作而创建了超出令牌列表范围的索引。您可以改为执行以下操作：

import spacy

nlp = spacy.load("en_core_web_sm")

listToStr = 'degeneration agents alpha alternative amd analysis angiogenesis anti anti vegf appears associated based best bevacizumab blindness blood'

simi = []
tokens = nlp(listToStr) 

for idx, tok in enumerate(tokens):
    sim = []
    for nextok in tokens[idx:]:
        sim.append(tok.similarity(nextok))
    simi.append(sim)

此测试每个单词与句子中下一个单词的相似性，因此结果是列表列表。