使用火花的Kudu连接问题

问题描述 投票:0回答:1

[当前,我正在一个项目中,我需要使用spark读取Kudu表。当我尝试使用spark-shell读取Kudu表时,我能够读取。但是,当我创建应用程序jar并使用spark2-submit选项提交相同的文件时,它给了我以下错误。我们的集群被kerberized。您能否在以下问题上帮助我。

Spark版本:2.4Kudu版本:1.6Scala版本:2.11

命令:

spark2-submit --master yarn --deploy-mode cluster \ 
              --class org.xxx.xxx.xxx \
              --jars "/home/xxx/geospark-1.2.0.jar,/home/xxx/geospark/geospark-sql_2.3-1.2.0.jar,/home/xxx/geospark/geospark-viz_2.3-1.2.0.jar,/home/xxx/geospark/kudu-spark2_2.11-1.9.0-cdh6.2.0.jar" \ 
              /home/xxx/xxx-0.0.1-SNAPSHOT.jar 
              --principal [email protected] \
              --keytab /home/xxx/user.keytab

输出:

20/04/02 17:06:56 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
20/04/02 17:06:56 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
20/04/02 17:06:59 ERROR yarn.ApplicationMaster: User class threw exception: java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:360)
    at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:145)
    at org.apache.kudu.spark.kudu.KuduRelation.<init>(DefaultSource.scala:256)
    at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:109)
    at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:87)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
    at org.apache.kudu.spark.kudu.package$KuduDataFrameReader.kudu(package.scala:32)
    at org.apache.GFence.GeoFence$.main(GeoFence.scala:47)
    at org.apache.GFence.GeoFence.main(GeoFence.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)```


apache-spark kudu
1个回答
0
投票

xxx.xxx.xxx.xxx:7051?您可以尝试对其进行修改

© www.soinside.com 2019 - 2024. All rights reserved.