以下是我的火花提交
spark2-submit --class my.class \
--master yarn \
--deploy-mode cluster \
--queue queue-name\
--executor-memory 10G \
--driver-memory 20G \
--num-executors 60 \
--conf spark.executor.memoryOverhead=4G \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.dynamicAllocation.maxExecutors=480 \
$HOME/myjar.jar param1 param2 param3
错误
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 50 in stage 27.0 failed 4 times,
most recent failure: Lost task 50.4 in stage 27.0 (TID 20899, cdts13hdfc07p.rxcorp.com, executor 962):
ExecutorLostFailure (executor 962 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits.
15.7 GB of 14 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
我的问题
您已经为每个 Spark 执行器分配了 10GB,您需要确保运行执行器的机器/节点上有足够的资源来满足其其他需求
容器的内存是
memory
+ memoryOverhead
的总和。
所以,你的执行器容器的内存是 10G + 4G = 14G