Azure DataFactory集成HDInsight Spark问题

问题描述 投票:0回答:1

当我使用Data Factory创建HDInsight Spark activity时,类型(脚本和jar)中有两个选项,当我选择要运行的jar(我使用scala开发jar文件时,它运行了成功 [remote livy submit),它失败,并且日志显示:

*stdout: 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.3009-43/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.3009-43/spark_llap/spark-llap-assembly-1.0.0.2.6.5.3009-43.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: --py-files given but primary resource is not a Python script
Run with --help for usage help or --verbose for debug output

stderr: 

YARN Diagnostics: 
java.lang.Exception: No YARN application is found with tag livy-batch-4-tpqnpjdw in 120 seconds. Please check your cluster status, it is may be very busy.
org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:236) scala.Option.getOrElse(Option.scala:120) org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:236) org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
*

但是当我选择脚本运行(使用python开发脚本时),它[[成功,有人可以解决这个问题吗?

apache-spark hdinsight azure-data-factory-2
1个回答
0
投票
这是HDInsight Spark活动中的预期行为。
© www.soinside.com 2019 - 2024. All rights reserved.