火花提交:使用jar无法找到表或视图

问题描述 投票:2回答:1

当我从intellij ide运行HiveRead.java时,我可以成功运行并获得结果。然后我创建了jar文件(这是一个maven项目),然后尝试从IDE运行,它给了我

ClassLoaderResolver for class "" gave error on creation : {1}

然后,我查看了SO的答案,发现我必须添加datanulcues jar,我做了类似的事情

java -jar /home/saurab/sparkProjects/spark_hive/target/myJar-jar-with-dependencies.jar --jars jars/datanucleus-api-jdo-3.2.6.jar,jars/datanucleus-core-3.2.10.jar,jars/datanucleus-rdbms-3.2.9.jar,/home/saurab/hadoopec/hive/lib/mysql-connector-java-5.1.38.jar

然后我收到此错误

org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.

[我发现应该在spark-submit的某个地方。所以我确实是这样

./bin/spark-submit --class HiveRead --master yarn  --jars jars/datanucleus-api-jdo-3.2.6.jar,jars/datanucleus-core-3.2.10.jar,jars/datanucleus-rdbms-3.2.9.jar,/home/saurab/hadoopec/hive/lib/mysql-connector-java-5.1.38.jar --files /home/saurab/hadoopec/spark/conf/hive-site.xml /home/saurab/sparkProjects/spark_hive/target/myJar-jar-with-dependencies.jar

现在我得到了新的错误类型

Table or view not found: `bigmart`.`o_sales`; 

帮我! :)

我已将我的hive-site.xml复制到/spark/conf,启动了蜂巢式元存储服务(hiveserver2 --service metastore]

如果有人感兴趣,这里是HiveRead.Java code

java hadoop apache-spark apache-spark-sql spark-submit
1个回答
0
投票

Spark会话无法读取配置单元目录。

使用如下的spark-submit命令提供hive-site.xml文件路径。

对于hortonworks-文件路径/usr/hdp/current/spark2-client/conf/hive-site.xml

在spark-submit命令中作为--files /usr/hdp/current/spark2-client/conf/hive-site.xml传递。

© www.soinside.com 2019 - 2024. All rights reserved.