标记返回字母而不是单词

Question

我尝试按照下面中页中的方法标记我的文本。运行代码后，令牌返回的是字母。你能帮忙让它返回单词吗？ HHC

https://odsc.medium.com/using-text-features-to-predict-the-great-stock-market-crash-of-1929-ebffa5a02097

import pysentiment2 as ps
import pandas as pd
import matplotlib.pyplot as plt

texts = 'why it return letter instead of words'

# Instantiate tokenizer for LM dictionary.
lm = ps.LM()
# Tokenize texts.
tokens = [lm.tokenize(t) for t in texts]
# Compute sentiment for each document.
sentiment = [lm.get_score(p)['Polarity'] for p in tokens]
print(sentiment)
tokens

Answer 1

无需过多查看代码并对其进行测试，

texts

可能应该是句子列表，因此请尝试以下操作：

texts = ['why it return letter instead of words']