使用Gensim训练Word2vec模型

问题描述 投票:0回答:1

这是我的code.it从excel文件(rev列)读取评论并列出列表。

xp是这样的

["['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],[ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']"]

但是当使用list for model时,它给出了错误“TypeError:'float'对象不可迭代”。我不知道我的问题在哪里。谢谢。

xp=[]
import gensim 
import logging
import pandas as pd 
file = r'FileNamelast.xlsx'
df = pd.read_excel(file,sheet_name='FileNamex')
pages = [i for i in range(0,1000)]


for page in  pages:

 text =df.loc[page,["rev"]]
 xp.append(text[0])


model = gensim.models.Word2Vec (xp, size=150, window=10, min_count=2, 
workers=10)
model.train(xp,total_examples=len(xp),epochs=10)

这就是我得到的.TypeError:'float'对象不可迭代

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-32-aa34c0e432bf> in <module>()
     14 
     15 
---> 16 model = gensim.models.Word2Vec (xp, size=150, window=10, min_count=2, workers=10)
     17 model.train(xp,total_examples=len(xp),epochs=10)

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in __init__(self, sentences, corpus_file, size, alpha, window, min_count, max_vocab_size, sample, seed, workers, min_alpha, sg, hs, negative, ns_exponent, cbow_mean, hashfxn, iter, null_word, trim_rule, sorted_vocab, batch_words, compute_loss, callbacks, max_final_vocab)
    765             callbacks=callbacks, batch_words=batch_words, trim_rule=trim_rule, sg=sg, alpha=alpha, window=window,
    766             seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean, min_alpha=min_alpha, compute_loss=compute_loss,
--> 767             fast_version=FAST_VERSION)
    768 
    769     def _do_train_epoch(self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\base_any2vec.py in __init__(self, sentences, corpus_file, workers, vector_size, epochs, callbacks, batch_words, trim_rule, sg, alpha, window, seed, hs, negative, ns_exponent, cbow_mean, min_alpha, compute_loss, fast_version, **kwargs)
    757                 raise TypeError("You can't pass a generator as the sentences argument. Try an iterator.")
    758 
--> 759             self.build_vocab(sentences=sentences, corpus_file=corpus_file, trim_rule=trim_rule)
    760             self.train(
    761                 sentences=sentences, corpus_file=corpus_file, total_examples=self.corpus_count,

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\base_any2vec.py in build_vocab(self, sentences, corpus_file, update, progress_per, keep_raw_vocab, trim_rule, **kwargs)
    934         """
    935         total_words, corpus_count = self.vocabulary.scan_vocab(
--> 936             sentences=sentences, corpus_file=corpus_file, progress_per=progress_per, trim_rule=trim_rule)
    937         self.corpus_count = corpus_count
    938         self.corpus_total_words = total_words

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in scan_vocab(self, sentences, corpus_file, progress_per, workers, trim_rule)
   1569             sentences = LineSentence(corpus_file)
   1570 
-> 1571         total_words, corpus_count = self._scan_vocab(sentences, progress_per, trim_rule)
   1572 
   1573         logger.info(

C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\word2vec.py in _scan_vocab(self, sentences, progress_per, trim_rule)
   1552                     sentence_no, total_words, len(vocab)
   1553                 )
-> 1554             for word in sentence:
   1555                 vocab[word] += 1
   1556             total_words += len(sentence)

TypeError: 'float' object is not iterable
python-3.x gensim word2vec
1个回答
1
投票

sentencesWord2Vec语料库参数应该是一个可迭代的单词列表序列。

您报告的xp值实际上是一个包含一个长字符串的列表:

[
  "['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],[ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']"
]

我不知道这会给你报告的错误,但这肯定是错的,所以应该修复。您应该在实例化xp之前打印Word2Vec,以确保您知道它包含的内容。

一个真正的列表,每个项目都是一个字符串标记列表,可以工作。因此,如果xp是以下是正确的:

    [
      ['intrepid', 'bumbling', 'duo', 'deliver', 'good', 'one'],
      ['better', 'offering', 'considerable', 'cv', 'freshly', 'qualified', 'private', 'investigator', 'thrust', 'murder', 'investigation', 'invisible'],
      [ 'man', 'alone', 'tell', 'fun', 'flow', 'decent', 'clip', 'need', 'say', 'sequence', 'comedy', 'gold', 'like', 'scene', 'restaurant', 'excellent', 'costello', 'pretending', 'work', 'ball', 'gym', 'final', 'reel']
    ]

但请注意:

  • Word2Vec对玩具大小的数据集表现不佳。因此,虽然这个微小的设置可能有助于检查基本的语法/格式问题,但是在您使用数十万个单词进行训练之前,不要期望得到真实的结果。
  • 如果您已经在实例化时提供了语料库,则无需调用train()。该模型将自动执行所有步骤。 (另一方面,如果您不提供语料库,则必须同时调用build_vocab()train()。)如果在INFO级别启用日志记录,幕后发生的所有步骤将更加清晰。
© www.soinside.com 2019 - 2024. All rights reserved.