ValueError：np.nan是无效文档，预期字节或unicode字符串

Question

我正在尝试对Uber-Review进行情绪分析。我使用了Naive bays sklearn来执行情绪分析，我使用了来自kaggle的修复数据，但测试数据是在xlsx表中，我使用pandas来创建数据框，

import pandas as pd
test=pd.read_excel("uber.xlsx",sep="\t",encoding="ISO-8859-1");
test.head(3)

当它返回d：type对象时，我使用它将其转换为list

test_text = []
for comments in comments_t:
    test_text.append(comments) 

我根据训练数据对文本进行分类的代码：

# Training Phase
from sklearn.naive_bayes import BernoulliNB
classifier = BernoulliNB().fit(train_documents,labels)

def sentiment(word):
    return classifier.predict(count_vectorizer.transform([word]))

但在预测时会返回此值错误：

/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in transform(self, raw_documents)
   1084 
   1085         # use the same matrix-building strategy as fit_transform
-> 1086         _, X = self._count_vocab(raw_documents, fixed_vocab=True)
   1087         if self.binary:
   1088             X.data.fill(1)

/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
    940         for doc in raw_documents:
    941             feature_counter = {}
--> 942             for feature in analyze(doc):
    943                 try:
    944                     feature_idx = vocabulary[feature]

/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in <lambda>(doc)
    326                                                tokenize)
    327             return lambda doc: self._word_ngrams(
--> 328                 tokenize(preprocess(self.decode(doc))), stop_words)
    329 
    330         else:

/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in decode(self, doc)
    141 
    142         if doc is np.nan:
--> 143             raise ValueError("np.nan is an invalid document, expected byte or "
    144                              "unicode string.")
    145 

ValueError: np.nan is an invalid document, expected byte or unicode string.

我试图按照这个解决：

https://stackoverflow.com/questions/39303912/tfidfvectorizer-in-scikit-learn-valueerror-np-nan-is-an-invalid-document

Answer 1

我在Uber的Kaggle找到的数据是https://www.kaggle.com/purvank/uber-rider-reviews-dataset/downloads/Uber_Ride_Reviews.csv/2

现在来解决你的问题

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import BernoulliNB

df = pd.read_csv('Uber_Ride_Reviews.csv')
df.head()
Out[7]: 
                                     ride_review    ...      sentiment
0  I completed running New York Marathon requeste...    ...              0
1  My appointment time auto repairs required earl...    ...              0
2  Whether I using Uber ride service Uber Eats or...    ...              0
3  Why hard understand I trying retrieve Uber cab...    ...              0
4  I South Beach FL I staying major hotel ordered...    ...              0


df.columns
Out[8]: Index(['ride_review', 'ride_rating', 'sentiment'], dtype='object')

vect  = CountVectorizer()
vect1 = vect.fit_transform(df['ride_review'])

classifier = BernoulliNB()
classifier.fit(vect1,df['sentiment'])

# predicting new comment it is giving O/p
new_test_= vect.transform(['uber ride is very good']) 
classifier.predict(new_test_)
Out[5]: array([0], dtype=int64)

 # but when applying your function sentiment you are only passing word, you need to 
 #passclassifier as well as Countvectorizer instance 

def sentiment(word, classifier, vect):
    return classifier.predict(vect.transform([word]))

#calling above function for new sentiment
sentiment('uber ride is very good', vect, classifier)
O/p --> Out[10]: array([0], dtype=int64)