如何文本功能和分类功能结合在Python?

问题描述 投票:1回答:1

我试图建立一个管道进行改造,分别编码的文本和类别特征,并结合他们被送入一个分类。目前,我有以下类别来选择数据:

class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        print(X[self.attribute_names].head())
        return X[self.attribute_names]

然后使用我有以下FeatureUnion与管道结合:

preprocessing = FeatureUnion([
    ("text_pipeline", Pipeline([
        ("select_text", DataFrameSelector(text_features)),
        ("count_vect", CountVectorizer()),
        ("word_count_to_vector", TfidfTransformer()),
    ])),
    ("cat_pipeline", Pipeline([
        ("select_cat", DataFrameSelector(cat_features)),
        ("cat_encoder", OneHotEncoder(sparse=False)),

    ])),
])

当执行full_pipeline.fit_transform(X_train)我收到以下错误:

ValueError                                Traceback (most recent call last)
<ipython-input-69-6927adc0ed62> in <module>()
     22 ])
     23 
---> 24 full_pipeline.fit_transform(X_train)

/anaconda3/lib/python3.6/site-packages/sklearn/pipeline.py in fit_transform(self, X, y, **fit_params)
    298         Xt, fit_params = self._fit(X, y, **fit_params)
    299         if hasattr(last_step, 'fit_transform'):
--> 300             return last_step.fit_transform(Xt, y, **fit_params)
    301         elif last_step is None:
    302             return Xt

/anaconda3/lib/python3.6/site-packages/sklearn/pipeline.py in fit_transform(self, X, y, **fit_params)
    798         self._update_transformer_list(transformers)
    799         if any(sparse.issparse(f) for f in Xs):
--> 800             Xs = sparse.hstack(Xs).tocsr()
    801         else:
    802             Xs = np.hstack(Xs)

/anaconda3/lib/python3.6/site-packages/scipy/sparse/construct.py in hstack(blocks, format, dtype)
    462 
    463     """
--> 464     return bmat([blocks], format=format, dtype=dtype)
    465 
    466 

/anaconda3/lib/python3.6/site-packages/scipy/sparse/construct.py in bmat(blocks, format, dtype)
    583                                                     exp=brow_lengths[i],
    584                                                     got=A.shape[0]))
--> 585                     raise ValueError(msg)
    586 
    587                 if bcol_lengths[j] == 0:

ValueError: blocks[0,:] has incompatible row dimensions. Got blocks[0,1].shape[0] == 1, expected 19634.

我想不出什么我做错了什么。任何帮助表示赞赏。

python machine-learning scikit-learn pipeline data-processing
1个回答
0
投票

所以我把它用hstack从scipy.sparse到Concat的两个稀疏矩阵的工作。请参见下面的代码:

from scipy.sparse import coo_matrix, hstack
from sklearn.preprocessing import OneHotEncoder
with_prod_tfidf = text_pipeline.fit_transform(with_prod['Text'])

#as per https://stackoverflow.com/questions/19710602/concatenate-sparse-matrices-in-python-using-scipy-numpy
with_prod_all = hstack([with_prod_tfidf, OneHotEncoder().fit_transform(with_prod[cat_features])])
print(with_prod_all.shape)
© www.soinside.com 2019 - 2024. All rights reserved.