Spark Executor 连接正在接近

问题描述 投票:0回答:0

您好,我正在阅读 parquet 文件,但在 AWS EMR 中遇到以下错误:-

错误 TransportResponseHandler:当来自 ip-100-73-112-181.ec2.internal/100.73.112.181:7337 的连接关闭时,仍有 518 个请求未完成

我的火花命令:-

spark-submit --deploy-mode cluster --class com.abcMain --driver-memory 6g --executor-memory 80g --conf avro.mapred.ignore.inputs.without.extension=false --conf "spark. driver.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=2 -XX:+UseCompressedOops" --conf spark.executor.cores=16 --conf "spark.executor.extraJavaOptions=-XX: +UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=2 -XX:+UseCompressedOops" --conf Spark.task.maxFailures=20 --conf Spark.ui.retainedTasks=1280 --conf Spark.yarn.am.attemptFailuresValidityInterval =1h --conf Spark.yarn.executor.failuresValidityInterval=1h --conf spark.yarn.max.executor.failures=100 --conf spark.yarn.maxAppAttempts=1

java apache-spark pyspark apache-spark-sql amazon-emr
© www.soinside.com 2019 - 2024. All rights reserved.