并行运行多个Spark应用程序

Question

我正在运行YARN + spark 2.4.5集群。我正在使用docker作为yarn client模式运行作业。我配置了主服务器和泊坞窗客户端，如下所示：

yarn-site.xml

<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>
 <property>
    <name>yarn.scheduler.fair.preemption</name>
    <value>true</value>
  </property>


  <property>
    <name>yarn.scheduler.fair.assignmultiple</name>
    <value>true</value>
  </property>

fairscheduler.xml：

<allocations>
  <pool name="default">
    <schedulingMode>FAIR</schedulingMode>
    <maxRunningApps>2</maxRunningApps>
    <fairSharePreemptionThreshold>0.6</fairSharePreemptionThreshold>
    <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
    <weight>2</weight>
    <minShare>2</minShare>
  </pool>

  <pool name="today">
    <schedulingMode>FAIR</schedulingMode>
    <maxRunningApps>2</maxRunningApps>
    <fairSharePreemptionThreshold>0.6</fairSharePreemptionThreshold>
    <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
    <weight>2</weight>
    <minShare>2</minShare>
  </pool>

  <pool name="yesterday">
    <schedulingMode>FAIR</schedulingMode>
    <maxRunningApps>2</maxRunningApps>
    <weight>1</weight>
    <fairSharePreemptionThreshold>0.4</fairSharePreemptionThreshold>
    <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
    <minShare>2</minShare>
  </pool>
</allocations>

现在，我在两个不同的容器中为两个spark应用程序共进午餐。一分钟或两分钟后，第一个然后第二个：

spark-submit --deploy-mode client --name job1 --queue today --py-files gs://babadod/spark.zip gs://babadod/spark/main.py --job job1 --job-args date=2020-04-02

spark-submit --deploy-mode client --name job2 --queue yesterday --py-files gs:/babadod/spark.zip gs://babadod/spark/main.py --job job2 --job-args date=2020-04-01

每个输出的开始：

20/04/02 13:52:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://namenode:4040
20/04/02 13:52:08 INFO FairSchedulableBuilder: Creating Fair Scheduler pools from /opt/spark/conf/fairscheduler.xml
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: default, schedulingMode: FAIR, minShare: 2, weight: 2
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: today, schedulingMode: FAIR, minShare: 2, weight: 2
20/04/02 13:52:09 INFO FairSchedulableBuilder: Created pool: yesterday, schedulingMode: FAIR, minShare: 2, weight: 1
20/04/02 13:52:09 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
20/04/02 13:52:09 INFO RMProxy: Connecting to ResourceManager at namenode/10.128.15.208:8032
20/04/02 13:52:09 INFO Client: Requesting a new application from cluster with 2 NodeManagers

第一个应用程序占用了所有可用资源，第二个应用程序在第一个完成后启动。我想念什么？谢谢

---

编辑

spark-defaults.conf：

spark.master yarn;
spark.dynamicAllocation.enabled true;
spark.shuffle.service.enabled true;
spark.yarn.shuffle.stopOnFailure false;

Answer 1

我找不到源，但正如我回想的那样，spark需要处于动态资源分配模式，以允许纱线在需要时杀死执行程序。

尝试通过以下设置启用它：https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation

（（您还需要设置随机播放服务：https://spark.apache.org/docs/latest/running-on-yarn.html#configuring-the-external-shuffle-service）