如何在LDA模型中动态传递用户提供的.txt文档？我已经尝试了下面的代码，但是无法提供适当的文档主题。我的.txt的主题与Sports相关，因此应将主题名称命名为Sports。它的输出为：
Score: 0.5569453835487366 - Topic: 0.008*"bike" + 0.005*"game" + 0.005*"team" + 0.004*"run" + 0.004*"virginia"
Score: 0.370819091796875 - Topic: 0.016*"game" + 0.014*"team" + 0.011*"play" + 0.008*"hockey" + 0.008*"player"
Score: 0.061239391565322876 -Topic: 0.010*"card" + 0.010*"window" + 0.008*"driver" + 0.007*"sale" + 0.006*"price"*
data = df.content.values.tolist()
data = [re.sub('\S*@\S*\s?', '', sent) for sent in data]
data = [re.sub('\s+', ' ', sent) for sent in data]
data = [re.sub("\'", "", sent) for sent in data]
def sent_to_words(sentences):
for sentence in sentences:
yield(gensim.utils.simple_preprocess(str(sentence), deacc=True)) # deacc=True removes punctuations
data_words = list(sent_to_words(data))
bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.
trigram = gensim.models.Phrases(bigram[data_words], threshold=100)
bigram_mod = gensim.models.phrases.Phraser(bigram)
trigram_mod = gensim.models.phrases.Phraser(trigram)
def remove_stopwords(texts):
return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]
def make_bigrams(texts):
return [bigram_mod[doc] for doc in texts]
def make_trigrams(texts):
return [trigram_mod[bigram_mod[doc]] for doc in texts]
def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
texts_out = []
for sent in texts:
doc = nlp(" ".join(sent))
texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
return texts_out
# Remove Stop Words
data_words_nostops = remove_stopwords(data_words)
# Form Bigrams
data_words_bigrams = make_bigrams(data_words_nostops)
data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
id2word = gensim.corpora.Dictionary(data_lemmatized)
texts = data_lemmatized
corpus = [id2word.doc2bow(text) for text in texts]
# Build LDA model
lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
id2word=id2word,
num_topics=20,
random_state=100,
update_every=1,
chunksize=100,
passes=10,
alpha='auto',
per_word_topics=True)
#f = io.open("text.txt", mode="r", encoding="utf-8")
p=open("text.txt", "r") #document by the user which is related to sports
if p.mode == 'r':
content = p.read()
bow_vector = id2word.doc2bow(lemmatization(p))
for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):
print("Score: {}\t Topic: {}".format(score, lda_model.print_topic(index, 5)))
您的所有代码都是正确的，但是我认为您对LDA建模的期望可能会有所下降。您收到的输出是正确的输出！
首先，您使用短语“主题名称”； LDA生成的主题没有名称，并且没有简单的映射到用于训练模型的数据标签。这是一个无监督的模型，通常您会使用没有标签的数据来训练LDA。如果您的语料库包含属于A，B，C，D类的文档，并且您训练了LDA模型以输出四个主题L，M，N，O，则不会遵循该规则，因为存在某些映射，例如：
A -> M
B -> L
C -> O
D -> N
第二，请注意输出中标记和主题之间的区别。 LDA的输出看起来像：
主题1：0.5-0.005 *“令牌_13” + 0.003 *“令牌_204” + ...
主题2：0.07-0.01 *“ token_24” + 0.001 *“ token_3” + ...
换句话说，每个文档都有属于每个主题的概率。每个主题都由以某种方式加权以唯一定义主题的每个语料标记的总和组成。
倾向于查看每个主题中权重最高的标记并将这些主题解释为一类。例如：
# If you have:
topic_1 = 0.1*"dog" + 0.08*"cat" + 0.04*"snake"
# It's tempting to name topic_1 = pets
但是这很难验证，并且在很大程度上取决于人类的直觉。 LDA的一种更常见用法是当您没有标签时，并且您想要确定哪些文档在语义上彼此相似，而不必确定文档的正确类标签是什么。
经过一番尝试后，这对我有用，如果有其他不同之处，请发表评论。
bow_vector = dictionary.doc2bow(preprocess(content))
q= lda_model[bow_vector]
from operator import itemgetter
res = max(q, key = itemgetter(1))[0]
res1 = max(q, key = itemgetter(1))[1]
if (res == 1 ):
print("This .txt file is related to Politics/Government, Accuracy:",res1)
elif (res == 2) :
print("This .txt file is related to sports, Accuracy:",res1)
elif res==3:
print("This .txt file is related to Computer, Accuracy:",res1)
elif..... (so on)
else.