k倍交叉验证以使用pyspark调整回归树模型

Question

我正在尝试使用k倍交叉验证来调整pyspark中生成的回归树。但是，到目前为止，我无法将pyspark的CrossValidator与pyspark的DecisionTree.trainRegressor结合使用。这是相关的代码。

    (trainingData, testData) = data.randomSplit([0.7, 0.3])

    model = DecisionTree.trainRegressor(trainingData, categoricalFeaturesInfo={}, impurity='variance', maxDepth=5, maxBins=32)

然后如何将k折交叉验证应用于回归变量？

Answer 1

您可以尝试以下方法：

(trainingData, testData) = data.randomSplit([0.7, 0.3])

model = DecisionTree.trainRegressor(trainingData, categoricalFeaturesInfo={}, impurity='variance', numClasses=2)

paramGrid = ParamGridBuilder() \
    .addGrid(model.maxDepth, [4, 5, 6, 7]) \
    .addGrid(model.maxBins, [24, 28, 32, 36]) \
    .build()

crossval = CrossValidator(estimator=model,
                          estimatorParamMaps=paramGrid,
                          evaluator=BinaryClassificationEvaluator(),
                          numFolds=3)  

# Run cross-validation, and choose the best set of parameters.
cvModel = crossval.fit(training)