Scala-Ignite提取数据不起作用-线程“ main”中的异常java.lang.NoClassDefFoundError：org / apache / ignite / spark / IgniteDataFrameSettings $

问题描述 投票：0回答：1

我正在尝试使用Scala从ignite表访问数据。我已经使用pyspark在ignite表中设置了数据，并且可以确认它正在工作并且已经设置。这是我正在使用的scala代码

import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.configuration.CacheConfiguration
import org.apache.ignite.{Ignite, Ignition}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.ignite.spark.IgniteDataFrameSettings._
import org.apache.ignite.spark.{IgniteContext, IgniteRDD}


object Main {
  def main(args: Array[String]) = 
    {
      val spark = SparkSession.builder.appName("App").config("spark.master", "local").getOrCreate()

      val df = spark.read.format(FORMAT_IGNITE).option(OPTION_TABLE, "KEY" ).option(OPTION_CONFIG_FILE, "/opt/ignite/apache-ignite/examples/config/example-ignite.xml")
    }
}

我正在使用sbt将其打包为jar，然后使用spark-submit命令运行该应用程序。

这里是完整的StackTrace：

$ /usr/share/spark-2.3.0-bin-hadoop2.6/bin/spark-submit target/scala-2.11/testing_2.11-0.1.0-SNAPSHOT.jar 
2020-05-28 07:57:46 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-28 07:57:47 INFO  SparkContext:54 - Running Spark version 2.3.0
2020-05-28 07:57:47 INFO  SparkContext:54 - Submitted application: PyStage
2020-05-28 07:57:47 INFO  SecurityManager:54 - Changing view acls to: root
2020-05-28 07:57:47 INFO  SecurityManager:54 - Changing modify acls to: root
2020-05-28 07:57:47 INFO  SecurityManager:54 - Changing view acls groups to: 
2020-05-28 07:57:47 INFO  SecurityManager:54 - Changing modify acls groups to: 
2020-05-28 07:57:47 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissi
ons: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2020-05-28 07:57:48 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 43585.
2020-05-28 07:57:48 INFO  SparkEnv:54 - Registering MapOutputTracker
2020-05-28 07:57:48 INFO  SparkEnv:54 - Registering BlockManagerMaster
2020-05-28 07:57:48 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2020-05-28 07:57:48 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2020-05-28 07:57:48 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-4f5e4f96-3dcf-4757-a5b4-eb96e98ab10b
2020-05-28 07:57:48 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2020-05-28 07:57:48 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2020-05-28 07:57:48 INFO  log:192 - Logging initialized @4539ms
2020-05-28 07:57:48 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2020-05-28 07:57:48 INFO  Server:414 - Started @4753ms
2020-05-28 07:57:48 WARN  Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2020-05-28 07:57:48 WARN  Utils:66 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
2020-05-28 07:57:48 INFO  AbstractConnector:278 - Started ServerConnector@408b35bf{HTTP/1.1,[http/1.1]}{0.0.0.0:4042}
2020-05-28 07:57:48 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4042.
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@82c57b3{/jobs,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1e886a5b{/jobs/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@d816dde{/jobs/job,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6c451c9c{/jobs/job/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31c269fd{/stages,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@372b0d86{/stages/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47747fb9{/stages/stage,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e9658b5{/stages/stage/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a7b6f69{/stages/pool,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@20312893{/stages/pool/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@70eecdc2{/storage,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c41709a{/storage/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7db0565c{/storage/rdd,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54ec8cc9{/storage/rdd/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52eacb4b{/environment,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5528a42c{/environment/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a551a63{/executors,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a6f5124{/executors/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1edb61b1{/executors/threadDump,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ec2bf82{/executors/threadDump/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@cc62a3b{/static,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1fe8d51b{/,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@781e7326{/api,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@324dcd31{/jobs/job/kill,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@503d56b5{/stages/stage/kill,null,AVAILABLE,@Spark}
2020-05-28 07:57:48 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://6e4338526cac:4042
2020-05-28 07:57:49 INFO  SparkContext:54 - Added JAR file:/home/Documents/PyStage/Job-Manager/testing/target/scala-2.11/testing_2.11-0.1.0-SNAPSHOT.jar at spark://6e4338526cac:43585/jars/testing_2.11-0.1.0-SNAPSHOT.jar with timestamp 1590652669088
2020-05-28 07:57:49 INFO  Executor:54 - Starting executor ID driver on host localhost
2020-05-28 07:57:49 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43227.
2020-05-28 07:57:49 INFO  NettyBlockTransferService:54 - Server created on 6e4338526cac:43227
2020-05-28 07:57:49 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2020-05-28 07:57:49 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 6e4338526cac, 43227, None)
2020-05-28 07:57:49 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 6e4338526cac:43227 with 366.3 MB RAM, BlockManagerId(driver, 6e4338526cac, 43227, None)
2020-05-28 07:57:49 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 6e4338526cac, 43227, None)
2020-05-28 07:57:49 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 6e4338526cac, 43227, None)
2020-05-28 07:57:49 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@25c5e994{/metrics/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:50 INFO  SharedState:54 - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/Documents/PyStage/Job-Manager/testing/spark-warehouse/').
2020-05-28 07:57:50 INFO  SharedState:54 - Warehouse path is 'file:/home/Documents/PyStage/Job-Manager/testing/spark-warehouse/'.
2020-05-28 07:57:50 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ca1f591{/SQL,null,AVAILABLE,@Spark}
2020-05-28 07:57:50 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@551de37d{/SQL/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:50 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@997d532{/SQL/execution,null,AVAILABLE,@Spark}
2020-05-28 07:57:50 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@273842a6{/SQL/execution/json,null,AVAILABLE,@Spark}
2020-05-28 07:57:50 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@b558294{/static/sql,null,AVAILABLE,@Spark}
2020-05-28 07:57:51 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/ignite/spark/IgniteDataFrameSettings$
        at Main$.main(Main.scala:21)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.ignite.spark.IgniteDataFrameSettings$
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 12 more
2020-05-28 07:57:51 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2020-05-28 07:57:51 INFO  AbstractConnector:318 - Stopped Spark@408b35bf{HTTP/1.1,[http/1.1]}{0.0.0.0:4042}
2020-05-28 07:57:51 INFO  SparkUI:54 - Stopped Spark web UI at http://6e4338526cac:4042
2020-05-28 07:57:51 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2020-05-28 07:57:51 INFO  MemoryStore:54 - MemoryStore cleared
2020-05-28 07:57:51 INFO  BlockManager:54 - BlockManager stopped
2020-05-28 07:57:51 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2020-05-28 07:57:51 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2020-05-28 07:57:51 INFO  SparkContext:54 - Successfully stopped SparkContext
2020-05-28 07:57:51 INFO  ShutdownHookManager:54 - Shutdown hook called
2020-05-28 07:57:51 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-53a4999c-b458-4dc0-b6d6-af52135c304a
2020-05-28 07:57:51 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-35f14174-cf0b-43cd-8438-c61323f2bc3d

正在使用的版本：

Scala version - 2.11.0
Spark version - 2.3.0
Ignite version - 2.8.0
Sbt version - 1.3.3

[请在这里帮助我，我是这个问题的新手，不确定如何解决该问题。谢谢

scala apache-spark pyspark ignite
1个回答
0
投票

下载Spark-Ignite jar尝试以下任何选项

  1. 将其添加到Spark的lib文件夹中
  2. 将其添加到--jars命令的spark-submit选项中。
  3. 或在--driver-class-path命令的spark-submit选项中添加。
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.