dataset.collectAsList（）导致集群中的java.lang.ClassCastException

Question

[当我使用IntelliJ在Local中执行List<Row> rows = (List<Row>) dataset.collectAsList();时，我得到了结果，但是当在Cluster中运行时，我得到了以下错误。我在代码中使用UDF

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
        at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
        at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2292)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2210)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2286)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2210)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:490)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at java.lang.Thread.run(Thread.java:748)

有什么想法吗？还是需要任何其他详细信息？这是模式

StructType(StructField(Mar,StringType,false),
 StructField(DY,StringType,false),
 StructField(MB,StringType,false),
 StructField(Med,StringType,false),
 StructField(DS,StringType,false),
 StructField(dist,StringType,false),
 StructField(DL,DecimalType(36,2),false),
 StructField(GP28,IntegerType,false),
 StructField(GPHH,IntegerType,false),
 StructField(CP28,IntegerType,false),
 StructField(CPHH,IntegerType,false),
 StructField(I28,LongType,false),
 StructField(IHH,LongType,false),
 StructField(U28,IntegerType,false),
 StructField(UHH,IntegerType,false))

Answer 1

您正在尝试转换List MapPartitionsRDD，这就是问题所在。它表示您的错误。