如何解决 scala 中的无效外部类型错误

Question

我刚刚开始在 scala 中工作，我将下面的代码写入到

assert

，但它未能实现架构不匹配。我无法弄清楚其背后的原因。

下面是我写的代码-

object xyzz {
    val df = Seq(
    Row(10528.53994, 20232, "abc")
    )

}

val schema: StructType = StructType(
      Seq(
        StructField("AMOUNT", DoubleType),
        StructField("PERIOD", IntegerType),
        StructField("ID", StringType)
    )
)

val test = spark.createDataFrame(spark.sparkContext.parallelize(xyz.df), schema)

assert(
    test.where(col("ID") === lit("abc")).select(col("AMOUNT")).collect()(0) == 10528.53994
)

由于以下错误而失败 -

if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, amount), DoubleType, false) AS AMOUNT#500610

if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 14, period), IntegerType, false) AS PERIOD#500611

if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, ID), StringType, false), true, false, true) AS ID#500612


Cause: java.lang.RuntimeException: java.lang.Integer is not a valid external type for schema of double

请帮忙

Answer 1

这个

test.where(col("ID") === lit("abc")).select(col("AMOUNT")).collect()(0)

将返回

org.apache.spark.sql.Row

的对象。您的断言失败，因为正如预期的那样，该对象正在与 Double 值进行比较。

更改您的代码，如下所示。

assert(test.where(col("ID") === lit("abc")).select(col("AMOUNT")).as[Double].collect.headOption.getOrElse(0D) == 10528.53994)