如何在Spark独立集群中获取应用程序状态？

Question

bounty到期在4天内。该问题的答案有资格获得+ 50声望赏金。Gokulraj正在寻找规范答案。

根据官方Spark文档，我们可以使用spark-submit --master spark://IP-ADDRESS:PORT --status SUBMISSION_ID

但是，当我尝试使用相同的连接时，它无法建立连接。主URL和PORT有效，并且我的spark在具有默认配置的独立群集中运行。

spark-submit --master spark://10.xx.xx8.xx5:7077 --status driver-20191119160551-0003
19/11/19 17:45:10 WARN RestSubmissionClient: Unable to connect to server spark://10.xx.xx8.xx5:7077.
Exception in thread "main" org.apache.spark.deploy.rest.SubmitRestConnectionException: Unable to connect to server
        at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$requestSubmissionStatus$3.apply(RestSubmissionClient.scala:165)
        at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$requestSubmissionStatus$3.apply(RestSubmissionClient.scala:148)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at org.apache.spark.deploy.rest.RestSubmissionClient.requestSubmissionStatus(RestSubmissionClient.scala:148)
        at org.apache.spark.deploy.SparkSubmit.requestStatus(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:88)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.deploy.rest.SubmitRestConnectionException: No response from server
        at org.apache.spark.deploy.rest.RestSubmissionClient.readResponse(RestSubmissionClient.scala:285)
        at org.apache.spark.deploy.rest.RestSubmissionClient.org$apache$spark$deploy$rest$RestSubmissionClient$$get(RestSubmissionClient.scala:195)
        at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$requestSubmissionStatus$3.apply(RestSubmissionClient.scala:152)
        ... 11 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
        at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
        at scala.concurrent.Await$.result(package.scala:190)
        at org.apache.spark.deploy.rest.RestSubmissionClient.readResponse(RestSubmissionClient.scala:278)
        ... 13 more

Answer 1

也取决于Spark版本，但也许您可以尝试使用REST API curl http://10.xx.xx8.xx5:7077/v1/submissions/status/driver-20191119160551-0003

有关更有用的命令，您可以在这里进行检查：https://spark.apache.org/docs/latest/monitoring.html。您希望阅读与您的版本匹配的版本。

如果此URL链接即使在Spark版本文档中被提及仍然无效，则可能是由于应用程序本身引起的，如其他建议。

希望有帮助！