dataset.collectAsList（）导致spark 2.1.1中出现java.lang.ClassCastException

Question

当我在Spark 2.0.1中执行List<Row> rows = (List<Row>) dataset.collectAsList();时，我得到的结果很好，但是在2.1.1中我遇到了以下错误

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
        at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
        at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2292)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2210)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2286)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2210)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:490)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at java.lang.Thread.run(Thread.java:748)

有什么想法吗？还是需要任何其他详细信息？这是模式

StructType(StructField(Mar,StringType,false),
 StructField(DY,StringType,false),
 StructField(MB,StringType,false),
 StructField(Med,StringType,false),
 StructField(DS,StringType,false),
 StructField(dist,StringType,false),
 StructField(DL,DecimalType(36,2),false),
 StructField(GP28,IntegerType,false),
 StructField(GPHH,IntegerType,false),
 StructField(CP28,IntegerType,false),
 StructField(CPHH,IntegerType,false),
 StructField(I28,LongType,false),
 StructField(IHH,LongType,false),
 StructField(U28,IntegerType,false),
 StructField(UHH,IntegerType,false))

Answer 1

您正在尝试转换List MapPartitionsRDD，这就是问题所在。它表示您的错误。