我按照https://spark.apache.org/docs/2.1.0/quick-start.html上的Scala教程进行操作
我的scala文件
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp {
def main(args: Array[String]) {
val logFile = "/data/README.md" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println(s"Lines with a: $numAs, Lines with b: $numBs")
sc.stop()
}
}
和build.sbt
name := "Simple Project"
version := "1.0"
scalaVersion := "2.12.4"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0"
我成功运行了sbt package
(已经删除除了scala源代码和build.sbt之外的所有内容,然后再次运行sbt package
)
[info] Loading project definition from /home/cpu11453local/workspace/testspark_scala/project
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/my_name/workspace/testspark_scala/)
[info] Packaging /home/my_name/workspace/testspark_scala/target/scala-2.12/simple-project_2.12-1.0.jar ...
[info] Done packaging.
[success] Total time: 1 s, completed Nov 8, 2017 12:15:24 PM
但是,当我运行spark提交时
$SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] simple-project_2.12-1.0.jar
我收到了错误
java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
在gist上完全火花提交输出
正如@Alexey所说,将Scala版本更改为2.11修复了问题。
build.sbt
name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.11"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"
请注意,Scala版本必须与Spark匹配。看看artifactId,spark-core_2.11意味着它与scala 2.11兼容(无向后或向前兼容)
以下是 Spark/Scala online guide中显示的最新Spark 2.4.1发布示例的build.sbt条目:
name := "SimpleApp"
version := "1.0"
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.1"
虽然一切都在IntelliJ IDE中正常工作,但应用程序仍然失败,但有以下异常,
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
使用'sbt package'命令创建包并从命令行运行spark-submit后如下;
spark-submit -v --class SimpleApp --master local[*] target\scala-2.12\simpleapp_2.12-1.0.jar