java.lang.NoClassDefFoundError：SimpleApp.main中的org / apache / spark / streaming / flume / FlumeUtils（SimpleApp.java:61）

Question

对于项目要求，我试图使用spark示例中的spark来构建FlumUtils示例。我能够创建jar文件。但在尝试执行它时，我收到以下错误。有人可以帮我解决这个问题吗？

Error: application failed with exception
java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils
        at SimpleApp.main(SimpleApp.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.flume.FlumeUtils
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)




  I have included the below dependency as pom file



<dependency> <!-- Spark Flume dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-flume_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Core dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Steaming dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.10</artifactId>
  <version>1.2.1</version>

我使用spark-submit --class SimpleApp target / simple-project-1.0.jar运行相同的程序

任何人都可以帮助我吗？

Answer 1

如果您使用spark-submit运行作业..请确保使用--jars选项提供spark-streaming_2.10 jar

Answer 2

从https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume-assembly下载spark-streaming-flume-assembly_2.10.jar，并使用选项--jars spark-streaming-flume-assembly_2.10.jar运行spark-submit，应解决由于运行时产生的NoClassDefFound错误问题依赖解决问题。