ValueError：X每个样本有1709个特征;期待2444

Question

我正在使用此代码：

import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize
import re

使用TFIDF矢量化

from sklearn.feature_extraction.text import TfidfVectorizer
tv=TfidfVectorizer(max_df=0.5,min_df=2,stop_words='english')

加载数据文件

df=pd.read_json('train.json',orient='columns')
test_df=pd.read_json('test.json',orient='columns')

df['seperated_ingredients'] = df['ingredients'].apply(','.join)
test_df['seperated_ingredients'] = test_df['ingredients'].apply(','.join)

df['seperated_ingredients']=df['seperated_ingredients'].str.lower()
test_df['seperated_ingredients']=test_df['seperated_ingredients'].str.lower()

cuisines={'thai':0,'vietnamese':1,'spanish':2,'southern_us':3,'russian':4,'moroccan':5,'mexican':6,'korean':7,'japanese':8,'jamaican':9,'italian':10,'irish':11,'indian':12,'greek':13,'french':14,'filipino':15,'chinese':16,'cajun_creole':17,'british':18,'brazilian':19 }
df.cuisine= [cuisines[item] for item in df.cuisine]

进行预处理

ho=df['seperated_ingredients']
ho=ho.replace(r'#([^\s]+)', r'\1', regex=True)
ho=ho.replace('\'"',regex=True)

ho=tv.fit_transform(ho)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(ho,df['cuisine'],random_state=0)


from sklearn.linear_model import LogisticRegression
clf= LogisticRegression(penalty='l1')
clf.fit(X_train, y_train)
clf.score(X_test,y_test)

from sklearn.linear_model import LogisticRegression
clf1= LogisticRegression(penalty='l1')
clf1.fit(ho,df['cuisine'])

hs=test_df['seperated_ingredients']

hs=hs.replace(r'#([^\s]+)', r'\1', regex=True)
hs=hs.replace('\'"',regex=True)
hs=tv.fit_transform(hs)

ss=clf1.predict(hs) # this line is giving error.

在预测时获得上述错误。有谁知道我做错了什么？

Answer 1

您不应该重新使用tfidf-vectorizer，而是使用具有相同词汇表形状的相同矢量化程序来编码测试数据。有docs的方法描述：

fit_transform(raw_documents, y=None)
  Learn vocabulary and idf, return term-document matrix.
  This is equivalent to fit followed by transform, but more efficiently implemented.

transform(raw_documents, copy=True)
  Transform documents to document-term matrix.
  Uses the vocabulary and document frequencies (df) learned by fit (or fit_transform).

你有ValueError: X has 1709 features per sample; expecting 2444因为矢量化器改装了测试数据并且创建了新的词汇表，所以测试数据被编码成另一种形状的数组。只需使用print(len(tv.vocabulary_))检查第二次fit_transform之前和之后的词汇量大小。此外，tf-idf词汇表可能在重新安装期间重新排序。

ho=df['seperated_ingredients']
ho=ho.replace(r'#([^\s]+)', r'\1', regex=True)
ho=ho.replace('\'"',regex=True)
ho=tv.fit_transform(ho)

然后使用预先训练的tf-idf矢量化器来使用变换函数对数据进行编码：

hs=test_df['seperated_ingredients']
hs=hs.replace(r'#([^\s]+)', r'\1', regex=True)
hs=hs.replace('\'"',regex=True)
hs=tv.transform(hs)

转换使用相同的词汇表进行，因此输出数组具有正确的形状。