这个问题与以下内容完全相同:
可以通过使用nltk.tokenize删除一些不必要的停用词来标记字符串。但是,如何将包含停用词的短语标记为单个标记,同时删除其他停用词?
例如:
输入:特朗普是美国总统。
产出:['特朗普','美国总统']
我怎样才能得到刚刚删除'是'和第一个'''但不删除'of'和第二个'the'的结果?
您可以使用nltk的Multi-Word Expression Tokenizer,它允许将多字表达式合并为单个标记。您可以创建多字表达式的词典并向其添加条目,如下所示:
from nltk.tokenize import MWETokenizer
mwetokenizer = MWETokenizer([('President','of','the','United','States')], separator=' ')
mwetokenizer.add_mwe(('President','of','France'))
请注意,MWETokenizer将标记化文本列表作为输入,并对其进行重新标记。所以,首先将句子标记为例如。与word_tokenize()
,然后将其提供给MWETokenizer:
from nltk.tokenize import word_tokenize
sentence = "Trump is the President of the United States, and Macron is the President of France."
mwetokenized_sentence = mwetokenizer.tokenize(word_tokenize(sentence))
# ['Trump', 'is', 'the', 'President of the United States', ',', 'and', 'Macron', 'is', 'the', 'President of France', '.']
然后,过滤掉停用词以获得最终过滤的标记化句子:
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
filtered_sentence = [token for token in mwetokenizer.tokenize(word_tokenize(sentence)) if token not in stop_words]
print(filtered_sentence)
输出:
['Trump', 'President of the United States', ',', 'Macron', 'President of France', '.']