ChiSqSelector选择了错误的功能？

Question

我从我的this example Shell的文档中复制粘贴的Spark 2.3.0。

import org.apache.spark.ml.feature.ChiSqSelector
import org.apache.spark.ml.linalg.Vectors

val data = Seq(
  (7, Vectors.dense(0.0, 0.0, 18.0, 1.0), 1.0),
  (8, Vectors.dense(0.0, 1.0, 12.0, 0.0), 0.0),
  (9, Vectors.dense(1.0, 0.0, 15.0, 0.1), 0.0)
)

val df = spark.createDataset(data).toDF("id", "features", "clicked")

val selector = new ChiSqSelector()
  .setNumTopFeatures(1)
  .setFeaturesCol("features")
  .setLabelCol("clicked")
  .setOutputCol("selectedFeatures")

val selectorModel = selector.fit(df)
val result = selectorModel.transform(df)
result.show
+---+------------------+-------+----------------+
| id|          features|clicked|selectedFeatures|
+---+------------------+-------+----------------+
|  7|[0.0,0.0,18.0,1.0]|    1.0|          [18.0]|
|  8|[0.0,1.0,12.0,0.0]|    0.0|          [12.0]|
|  9|[1.0,0.0,15.0,0.1]|    0.0|          [15.0]|
+---+------------------+-------+----------------+

selectorModel.selectedFeatures
res2: Array[Int] = Array(2)

ChiSqSelector错误地选择了feature 2而不是feature 3（基于文档和常识，功能3应该是正确的）

Answer 1

Chi-Squared功能选择operates on categorical data

ChiSqSelector代表Chi-Squared功能选择。它使用具有分类特征的标记数据进行操作

因此两个特征同样好（尽管我们应该强调，即使用作连续变量，这两个特征也可用于推导平凡的完美分类器）。

import org.apache.spark.mllib.linalg.{Vectors => OldVectors}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.stat.Statistics

Statistics.chiSqTest(sc.parallelize(data.map { 
  case (_, v, l) => LabeledPoint(l, OldVectors.fromML(v)) 
})).slice(2, 4)

Array[org.apache.spark.mllib.stat.test.ChiSqTestResult] =
Array(Chi squared test summary:
method: pearson
degrees of freedom = 2
statistic = 3.0
pValue = 0.22313016014843035
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.., Chi squared test summary:
method: pearson
degrees of freedom = 2
statistic = 3.0000000000000004
pValue = 0.22313016014843035
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent..)

并且测试结果与其他工具一致。例如在R（used as a reference for selector tests）中：

y <- as.factor(c("1.0", "0.0", "0.0"))
x2 <- as.factor(c("18.0", "12.0", "15.0"))
x3 <- as.factor(c("1.0", "0.0", "0.1"))

chisq.test(table(x2, y))

    Pearson's Chi-squared test

data:  table(x2, y)
X-squared = 3, df = 2, p-value = 0.2231

Warning message:
In chisq.test(table(x2, y)) : Chi-squared approximation may be incorrect

chisq.test(table(x3, y))

    Pearson's Chi-squared test

data:  table(x3, y)
X-squared = 3, df = 2, p-value = 0.2231

Warning message:
In chisq.test(table(x3, y)) : Chi-squared approximation may be incorrect

选择器just sorts data by p-value和sortBy is stable，它是先到先得。如果切换功能的顺序，将选择另一个。