如何让天真的贝叶斯分类器工作？

Question

我尝试使用Naive Bayes分类器对我的样本语料库进行分类。示例语料库如下（存储在myfile.csv中）：

"Text";"label"
“There be no significant perinephric collection";"label1”
“There be also fluid collection”;”label2”
“No discrete epidural collection or abscess be see";"label1”
“This be highly suggestive of epidural abscess”;”label2”
“No feature of spondylodiscitis be see”;”label1”
“At the level of l2 l3 there be loculated epidural fluid collection”;”label2”

分类器的代码如下：

# libraries for dataset preparation, feature engineering, model training 
import pandas as pd
import csv
from sklearn import svm
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB

#Data preparation
data = pd.read_csv(open('myfile.csv'), sep=';', quoting=csv.QUOTE_NONE)

# Creating Bag of Words
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data)
print(X_train_counts.shape)

#From occurrences to frequencies
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
print(X_train_tf.shape)

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
print(X_train_tfidf.shape)

#Training a classifier
clf = MultinomialNB().fit(X_train_tfidf, data['label'])

#Predicting with the classifier
docs_new = ['there is no spondylodiscitis', 'there is a large fluid collection']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)
predicted = clf.predict(X_new_tfidf)
for doc, category in zip(docs_new, predicted): 
    print('%r => %s' % (doc, data['label']))

每当我尝试运行预测时，我都会收到以下错误：

KeyError: 'label'

我哪里错了？

Answer 1

如有疑问，请在REPL或调试器中加载代码。观察...中的任何内容与您的问题无关。

import pandas as pd
import csv
...

data = pd.read_csv(open('myfile.csv'), sep=';', quoting=csv.QUOTE_NONE)
import pdb; pdb.set_trace()
...

现在我们可以交互式地查询data对象：

(Pdb) data.keys()
Index(['"Text"', '"label"'], dtype='object')
(Pdb) data['"label"']
0    "label1”
1    ”label2”
2    "label1”
3    ”label2”
4    ”label1”
5    ”label2”
Name: "label", dtype: object
(Pdb) data["label"]
*** KeyError: 'label'

请注意，键是'"Test"'和'"label"'，而不是"Test"和"label"。所以你不能做data["label"]，或者你会得到你所看到的KeyError。你不得不说data['"label"']。

Answer 2

看起来你的数据有引号，为什么你在那里指定了QUOTE_NONE？

Answer 3

如果你想能够使用data['label']访问pandas列，你的第一行应该是：

Text;label

不是这个 ：

"Text";"label"

这样你就必须像这样索引你的标签col;

data['"label"']

哪个看起来不错