我在运行spark job时遇到错误。请为spark和cassandra连接器建议正确的版本。
下面是我的build.sbt
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-streaming" % "2.2.0-cdh6.0.1" % "provided",
"org.apache.spark" %% "spark-core" % "2.2.0-cdh6.0.1" % "provided", // excludeAll ExclusionRule(organization = "javax.servlet"),
"org.apache.spark" %% "spark-sql" % "2.2.0-cdh6.0.1" % "provided",
"org.apache.spark" %% "`enter code here`spark-streaming-kafka-0-10" % "2.2.0-cdh6.0.1",
"org.apache.hbase" % "hbase-client" % "2.0.0-cdh6.0.1",
"org.apache.hbase" % "hbase-common" % "2.0.0-cdh6.0.1",
"com.datastax.spark" %% "spark-cassandra-connector" % "2.0.10",
"net.liftweb" %% "lift-json" % "3.3.0",
"com.typesafe" % "config" % "1.2.1"
)
一旦提交了火花工作,我就会低于错误
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
at com.datastax.spark.connector.streaming.DStreamFunctions.saveToCassandra$default$4(DStreamFunctions.scala:47)
at com.StreamingPrerequisiteLoad$.main(StreamingPrerequisiteLoad.scala:72)
at com.StreamingPrerequisiteLoad.main(StreamingPrerequisiteLoad.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
我遇到了与spark-cassandra-connector类似的问题,所以它是如何工作的看起来,对于spark版本2.2和scala 11.8.0,spark-cassandra-connector 2.3.0将起作用。还要添加commons-configuration 1.9 version jar ,因为它会抛出异常NoClassDefFound:/ org / apache / commons / configuration / ConfigurationException。尝试使用以下依赖项:
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0" % "provided"
libraryDependencies += "net.liftweb" %% "lift-json" % "3.0.2"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.0" //% "provided"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.3.0" //% "provided"
libraryDependencies += "commons-configuration" % "commons-configuration" % "1.9" //% "provided"
assemblyMergeStrategy in assembly := {
case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
case x => (mergeStrategy in assembly).value(x)
}