类型错误：PySpark 中的 Xgboost 无法调用“JavaPackage”对象

Question

我正在尝试使 Scala Xgboost API 可用于我的 PySpark Notebook。并关注此博客： https://towardsdatascience.com/pyspark-and-xgboost-integration-tested-on-the-kaggle-titanic-dataset-4e75a568bdb 但是，继续遇到以下错误：

spark._jvm.ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator
<py4j.java_gateway.JavaPackage at 0x7fa650fe7a58>

from sparkxgb import XGBoostEstimator

xgboost = XGBoostEstimator(
    featuresCol="features", 
    labelCol="Survival", 
    predictionCol="prediction"
)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-1765fb9e3344> in <module>
      4     featuresCol="features",
      5     labelCol="Survival",
----> 6     predictionCol="prediction"
      7 )

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/__init__.py in wrapper(self, *args, **kwargs)
    108             raise TypeError("Method %s forces keyword arguments." % func.__name__)
    109         self._input_kwargs = kwargs
--> 110         return func(self, **kwargs)
    111     return wrapper
    112 

~/local/spark-3536cd7a-6188-4ca8-b3d0-57d42cd01531/userFiles-0a0d90bc-96b4-43f2-bf21-00ae0e6f7309/sparkxgb.zip/sparkxgb/xgboost.py in __init__(self, checkpoint_path, checkpointInterval, missing, nthread, nworkers, silent, use_external_memory, baseMarginCol, featuresCol, labelCol, predictionCol, weightCol, base_score, booster, eval_metric, num_class, num_round, objective, seed, alpha, colsample_bytree, colsample_bylevel, eta, gamma, grow_policy, max_bin, max_delta_step, max_depth, min_child_weight, reg_lambda, scale_pos_weight, sketch_eps, subsample, tree_method, normalize_type, rate_drop, sample_type, skip_drop, lambda_bias)
    113 
    114         super(XGBoostEstimator, self).__init__()
--> 115         self._java_obj = self._new_java_obj("ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator", self.uid)
    116         self._create_params_from_java()
    117         self._setDefault(

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
     65             java_obj = getattr(java_obj, name)
     66         java_args = [_py2java(sc, arg) for arg in args]
---> 67         return java_obj(*java_args)
     68 
     69     @staticmethod

TypeError: 'JavaPackage' object is not callable

我已经用谷歌搜索了这个错误并尝试了以下操作。我从这个博客中得到了所有想法https://github.com/JohnSnowLabs/spark-nlp/issues/232:

1. 确保 Xgboost4j 位于 SPARK_DIST_CLASSPATH 中。已经检查过了。

    $echo $SPARK_DIST_CLASSPATH |  tr " " "\n" | grep 'xgboost4j' | rev | cut -d'/' -f1 | rev
    xgboost4j-0.72.jar
    xgboost4j-spark.72.jar

2. 确保它们已添加到 EXTRA_CLASSPATH 中。 - 完成
3. 更新配置。

'export PYSPARK_SUBMIT_ARGS="--conf spark.jars=$SPARK_HOME/jars/* --conf spark.driver.extraClassPath=$SPARK_HOME/jars/* --conf spark.executor.extraClassPath=$SPARK_HOME/jars/* pyspark-shell"',

硬件信息：

• 机器：Linux
• 使用 Jupyter Notebook。
• Spark 版本 2.4.0
• python3.6

Answer 1

我发现了问题，问题是

sparkxbg.zip

（我通过互联网下载的）是为

xgboost4j-0.72

编写的。然而，我的罐子来自

xgoost4j-0.9

。并且API已经完全改变了。结果，0.9版本没有任何名为

ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator

的类。因此出现错误。

您可以在这里看到 API 的差异：release_0.72 与 v0.90