从列表中删除接近的匹配项/类似短语

Question

我正在努力删除列表中的类似短语，但遇到了一个小障碍。

我有句子和短语，短语与句子相关。句子的所有短语都在一个列表中。

让短语列表为：p=[['This is great','is great','place for drinks','for drinks'],['Tonight is a good','good night','is a good','for movies']]

我希望我的输出为[['This is great','place for drinks'],['Tonight is a good','for movies']]

基本上，我想获得列表中所有最长的唯一短语。

我看了Fuzzywuzzy库，但是无法找到一个好的解决方案。

这是我的代码：

def remove_dup(arr, threshold=80):
    ret_arr =[]
    for item in arr:
        if item[1]<threshold:
            ret_arr.append(item[0])
    return ret_arr

def find_important(sents=sents, phrase=phrase):

    import os, random
    from fuzzywuzzy import process, fuzz

    all_processed = [] #final array to be returned
    for i in range(len(sents)):

        new_arr = [] #reshaped phrases for a single sentence
        for item in phrase[i]:
            new_arr.append(item)

        new_arr.sort(reverse=True, key=lambda x : len(x)) #sort with highest length

        important = [] #array to store terms
        important = process.extractBests(new_arr[0], new_arr) #to get levenshtein distance matches
        to_proc = remove_dup(important) #remove_dup removes all relatively matching terms.
        to_proc.append(important[0][0]) #the term with highest match is obviously the important term.


        all_processed.append(to_proc) #add non duplicates to all_processed[]

    return all_processed

有人可以指出我所缺少的，或者有什么更好的方法吗？预先感谢！

Answer 1

我会使用每个短语与所有其他短语之间的差异。如果一个短语与所有其他短语相比至少有一个不同的单词，则它是唯一的，应予以保留。

我也使其完全匹配和添加空格变得健壮

sentences = [['This is great','is great','place for drinks','for drinks'],
['Tonight is a good','good night','is a good','for movies'],
['Axe far his favorite brand for deodorant body spray',' Axe far his favorite brand for deodorant spray','Axe is']]

new_sentences = []
s = " "
for phrases in sentences :
    new_phrases = []
    phrases = [phrase.split() for phrase in phrases]
    for i in range(len(phrases)) :
        phrase = phrases[i]
        if all([len(set(phrase).difference(phrases[j])) > 0 or i == j for j in range(len(phrases))]) :
            new_phrases.append(phrase)
    new_phrases = [s.join(phrase) for phrase in new_phrases]
    new_sentences.append(new_phrases)
print(new_sentences)

输出：

[['这很棒，'喝酒的地方']，

[[今晚真好，'晚安，'看电影']，

[[斧头是他最喜欢的除臭身体喷雾品牌，'斧头是]]]