如何使用Tf-idf功能训练模型？

Question

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf= True, 
                       min_df = 5, 
                       norm= 'l2', 
                       ngram_range= (1,2), 
                       stop_words ='english')

feature1 = tfidf.fit_transform(df.Rejoined_Stem)
array_of_feature = feature1.toarray()

我使用上面的代码来获取文本文档的功能。

from sklearn.naive_bayes import MultinomialNB # Multinomial Naive Bayes on Lemmatized Text
X_train, X_test, y_train, y_test = train_test_split(df['Rejoined_Lemmatize'], df['Product'], random_state = 0)
X_train_counts = tfidf.fit_transform(X_train)
clf = MultinomialNB().fit(X_train_counts, y_train)
y_pred = clf.predict(tfidf.transform(X_test))

然后，我使用此代码来训练我的模型。有人可以解释训练模型时上述特征的使用情况吗，因为在训练过程中没有在任何地方使用feature1变量？

从sklearn.feature_extraction.text导入TfidfVectorizer tfidf = TfidfVectorizer（sublinear_tf = True，min_df = 5，范数='l2'，...

Answer 1

否，执行另一个转换feature1时未使用X_train_count。

让我们以逻辑流程浏览代码，仅使用特征提取和模型训练中使用的变量。]​​>

# imports used
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

# split data random state 0 and test_size 0.25 default as you did not give the test_size

X_train, X_test, y_train, y_test = train_test_split(df[['Rejoined_Lemmatize']], df['Product'], random_state = 0)

# you initiated your transformer to `fit_transform` X_train, and `transform` X_test

tfidf = TfidfVectorizer(sublinear_tf= True, 
                       min_df = 5, 
                       norm= 'l2', 
                       ngram_range= (1,2), 
                       stop_words ='english')


X_train_counts = tfidf.fit_transform(X_train)
X_test_counts = tfidf.transform(X_test)

# you initiated your model and fit X_train_counts and y_train
clf = MultinomialNB()
cls.fit(X_train_counts, y_train)

# you predicted from your transformed features
y_pred = clf.predict(X_test_counts)