[当前,我正在一个项目中,我需要使用spark读取Kudu表。当我尝试使用spark-shell读取Kudu表时,我能够读取。但是,当我创建应用程序jar并使用spark2-submit
选项提交相同的文件时,它给了我以下错误。我们的集群被kerberized。您能否在以下问题上帮助我。
Spark版本:2.4Kudu版本:1.6Scala版本:2.11
命令:
spark2-submit --master yarn --deploy-mode cluster \
--class org.xxx.xxx.xxx \
--jars "/home/xxx/geospark-1.2.0.jar,/home/xxx/geospark/geospark-sql_2.3-1.2.0.jar,/home/xxx/geospark/geospark-viz_2.3-1.2.0.jar,/home/xxx/geospark/kudu-spark2_2.11-1.9.0-cdh6.2.0.jar" \
/home/xxx/xxx-0.0.1-SNAPSHOT.jar
--principal [email protected] \
--keytab /home/xxx/user.keytab
输出:
20/04/02 17:06:56 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
20/04/02 17:06:56 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
20/04/02 17:06:59 ERROR yarn.ApplicationMaster: User class threw exception: java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx.xxx.xxx.xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:145)
at org.apache.kudu.spark.kudu.KuduRelation.<init>(DefaultSource.scala:256)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:109)
at org.apache.kudu.spark.kudu.DefaultSource.createRelation(DefaultSource.scala:87)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at org.apache.kudu.spark.kudu.package$KuduDataFrameReader.kudu(package.scala:32)
at org.apache.GFence.GeoFence$.main(GeoFence.scala:47)
at org.apache.GFence.GeoFence.main(GeoFence.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)```
xxx.xxx.xxx.xxx:7051?您可以尝试对其进行修改