如何在Pyspark中设置KryoSerializer？

Question

我是Pyspark的新手，请帮助我：

spark = SparkSession.builder.appName("FlightDelayRDD").master("local[*]").getOrCreate()
sc = spark.sparkContext
sc.setSystemProperty("spark.dynamicAllocation.enabled", "true")
sc.setSystemProperty("spark.dynamicAllocation.initialExecutors", "6")
sc.setSystemProperty("spark.dynamicAllocation.minExecutors", "6")
sc.setSystemProperty("spark.dynamicAllocation.schedulerBacklogTimeout", "0.5s")
sc.setSystemProperty("spark.speculation", "true")

我想像上面配置的那样在pyspark中设置KryoSerializer。

Answer 1

尝试使用：

sc.setSystemProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer")