排查从 Jupyter Notebook 到 .py 文件的自定义管道类转换中的 OneHotEncoder 问题

Question

tl;dr：我在

ipynb

文件中定义了一个管道，工作正常，但是当我尝试将其封装在

Class

中时，它没有按预期工作。我可能在

OneHoteEncode

中犯了一些错误。由于代码的原因，问题有点长，但可能非常简单明了。

编辑：如果我运行

housing = housing[housing['ocean_proximity'] != "ISLAND"]

，代码运行不会出现错误。因此，问题确实属于这一类，这可能使我的问题变得非常简单。我再次检查了

OneHotEncode

，但我不知道它出了什么问题。

EDIT2：运行

grid_search.fit(housing.iloc[:100], housing_labels.iloc[:100])

不会出现错误。此外，

...housing.iloc[:1000]

也没有给出任何错误。但是在

[:5000]

上运行会得到

ValueError: The feature names should match those that were passed during fit. Feature names seen at fit time, yet now missing: cat__ocean_proximity_ISLAND

。 该类别在数据集中的示例很少。问题真的出在我如何进行 OneHotEncoding 上？

问题：

我有一个来自 Jupyter Notebook 中编写的练习的转换管道。它的工作没有错误。作为我自己的练习，我尝试以更专业的方式在不同的 python

.py

文件中编写函数、管道、模型等。但是，当在

Class

文件中将管道编写为 python

.py

并将其作为模块导入到主

.py

文件中时，有些东西无法正常工作。

笔记本上写的管道：

class ClusterSimilarity(BaseEstimator, TransformerMixin):
    def __init__(self, n_clusters=10, gamma=1.0, random_state=None):
        self.n_clusters = n_clusters
        self.gamma = gamma
        self.random_state = random_state

    def fit(self, X, y=None, sample_weight=None):
        self.kmeans_ = KMeans(self.n_clusters, n_init=10,
                              random_state=self.random_state)
        self.kmeans_.fit(X, sample_weight=sample_weight)
        return self  # always return self!

    def transform(self, X):
        return rbf_kernel(X, self.kmeans_.cluster_centers_, gamma=self.gamma)

    def get_feature_names_out(self, names=None):
        return [f"Cluster {i} similarity" for i in range(self.n_clusters)]

def column_ratio(X):
    return X[:, [0]] / X[:, [1]]

def ratio_name(function_transformer, feature_names_in):
    return ["ratio"]  # feature names out

def ratio_pipeline():
    return make_pipeline(
        SimpleImputer(strategy="median"),
        FunctionTransformer(column_ratio, feature_names_out=ratio_name),
        StandardScaler())

log_pipeline = make_pipeline(
    SimpleImputer(strategy="median"),
    FunctionTransformer(np.log, feature_names_out="one-to-one"),
    StandardScaler())
cat_pipeline = make_pipeline(
    SimpleImputer(strategy="most_frequent"),
    OneHotEncoder(handle_unknown="ignore"))
cluster_simil = ClusterSimilarity(n_clusters=10, gamma=1., random_state=42)
default_num_pipeline = make_pipeline(SimpleImputer(strategy="median"),
                                     StandardScaler())
preprocessing = ColumnTransformer([
        ("bedrooms", ratio_pipeline(), ["total_bedrooms", "total_rooms"]),
        ("rooms_per_house", ratio_pipeline(), ["total_rooms", "households"]),
        ("people_per_house", ratio_pipeline(), ["population", "households"]),
        ("log", log_pipeline, ["total_bedrooms", "total_rooms", "population",
                               "households", "median_income"]),
        ("geo", cluster_simil, ["latitude", "longitude"]),
        ("cat", cat_pipeline, make_column_selector(dtype_include=object)),
    ],
    remainder=default_num_pipeline)  # one column remaining: housing_median_age

以及我如何将其重写为 python 文件中的类：

from sklearn import set_config

set_config(transform_output='pandas')

class Preprocessor(TransformerMixin):
    def __init__(self):
        pass

    def fit(self, X, y=None):
        self._cluster_simil = ClusterSimilarity(n_clusters=10, gamma=1., random_state=42)
        return self

    def transform(self, X):
        preprocessing = self._preprocessing()
        return preprocessing.fit_transform(X)

    def _column_ratio(self, X):
        ratio = X.iloc[:, 0] / X.iloc[:, 1]
        return np.reshape(ratio.to_numpy(), (-1, 1))

    def _ratio_name(self, function_transformer, feature_names_in):
        return ["ratio"]  # feature names out

    def _ratio_pipeline(self):
        return make_pipeline(
            SimpleImputer(strategy="median"),
            FunctionTransformer(self._column_ratio, feature_names_out=self._ratio_name),
        StandardScaler()
    )

    def _log_pipeline(self):
        return make_pipeline(
            SimpleImputer(strategy="median"),
            FunctionTransformer(np.log, feature_names_out="one-to-one"),
            StandardScaler()
        )

    def _cat_pipeline(self):
        return make_pipeline(
            SimpleImputer(strategy="most_frequent"),
            OneHotEncoder(handle_unknown="ignore", sparse_output=False)
        )

    def _default_num_pipeline(self):
        return make_pipeline(SimpleImputer(strategy="median"),
                             StandardScaler()
        )

    def _preprocessing(self):
        return ColumnTransformer([
            ("bedrooms", self._ratio_pipeline(), ["total_bedrooms", "total_rooms"]),
            ("rooms_per_house", self._ratio_pipeline(), ["total_rooms", "households"]),
            ("people_per_house", self._ratio_pipeline(), ["population", "households"]),
            ("log", self._log_pipeline(), ["total_bedrooms", "total_rooms", "population",
                                "households", "median_income"]),
            ("geo", self._cluster_simil, ["latitude", "longitude"]),
            ("cat", self._cat_pipeline(), make_column_selector(dtype_include=object)),
    ],
    remainder=self._default_num_pipeline())  # one column remaining: housing_median_age

class ClusterSimilarity(BaseEstimator, TransformerMixin):
    def __init__(self, n_clusters=10, gamma=1.0, random_state=None):
        self.n_clusters = n_clusters
        self.gamma = gamma
        self.random_state = random_state

    def fit(self, X, y=None, sample_weight=None):
        self.kmeans_ = KMeans(self.n_clusters, n_init=10,
                              random_state=self.random_state)
        self.kmeans_.fit(X, sample_weight=sample_weight)
        return self  # always return self!

    def transform(self, X):
        return rbf_kernel(X, self.kmeans_.cluster_centers_, gamma=self.gamma)

    def get_feature_names_out(self, names=None):
        return [f"Cluster {i} similarity" for i in range(self.n_clusters)]

但它无法正常工作。当我打电话只是为了测试时：

preprocessor = Preprocessor()
X_train = preprocessor.fit_transform(housing)

X_train.info()

的输出正是它应该的样子。但是当我尝试

gridSearch

时：

svr_pipeline = Pipeline([("preprocessing", preprocessor), ("svr", SVR())])
grid_search = GridSearchCV(svr_pipeline, param_grid, cv=3,
                           scoring='neg_root_mean_squared_error')
                       
grid_search.fit(housing.iloc[:5000], housing_labels.iloc[:5000])

它输出警告：

ValueError: The feature names should match those that were passed during fit.
Feature names seen at fit time, yet now missing:
- cat__ocean_proximity_ISLAND

UserWarning: One or more of the test scores are non-finite:
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
 nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
 nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
 nan nan nan nan nan nan nan nan]

此警告在

GridSearch

的每一步都会重复，并且它指向的列始终是相同的

cat__ocean_proximity_ISLAND

，来自管道的

def _cat_pipeline(self):

部分，并且是使用

OneHotEncode

的结果

注意：上面的代码可以在我的笔记本上运行。事实上，它是《机器学习实践》一书中练习的解决方案。问题是当我尝试将作者在另一个 .py 文件中给出的代码重写为类时，以便我可以导入管道并使用它。因此，问题可能出在

class Preprocessor(TransformerMixin):

的某些行上，但我真的不知道在哪里。由于错误可能来自

OneHotEncode

，我认为我滥用了它，但我没有更改该部分的原始代码。我不知道如何修复它，也不知道为什么它说它在拟合中看到该列，但在预测中却看不到。如何修复并避免将来出现此类错误？

Answer 1

CV代表交叉验证。这意味着，为了评估单个网格点的性能（对于超参数的每种组合），需要针对三个不同的数据分割（即您在参数中传递的 cv=3）训练三个不同的模型，然后最终的准确度结果是所有结果的平均值。从概念上讲，该过程看起来像这样：

[第一个数据 1/3 (
• test

  )，第二个数据 1/3 (train)，第三个数据 1/3 (train)] -> 准确度 80%

[第一个数据 1/3 (
• train

  )，第二个数据 1/3 (test)，第三个数据 1/3 (train)] -> 准确度 82%

[第一个数据 1/3 (
• train

  )，第二个数据 1/3 (train)，第三个数据 1/3 (test)] -> 准确度 85%

预计总性能为(80+82+85)/3=82.33%。现在，我猜想通过你正在学习的书和你的代码水平你已经知道了。真正的问题在于

训练/测试分割是在管道开始时完成的

。这对于避免数据泄漏是有意义的，因为例如，如果您想通过平均值来估算某些 NA 值，您只想使用每个分割上的训练集数据的平均值，而不是整个数据，因为这会污染测试集的训练分割信息。 在one-hot编码中，您必须在分割之前执行编码。想象一下，当你在这种自动分裂中不走运时会发生什么。测试集中有一个标签没有出现在训练集中。训练集中缺少该标签，因此没有对其进行编码，因此现在将该编码应用于测试集会产生错误。应用之前的一个热编码（可能作为预管道）并在管道的其余部分上使用带有 CV 的网格搜索，这应该可以解决这个问题。