Presto + Spark 的 ThriftServer:null

问题描述 投票:0回答:0

我正在 Minio、Spark 和 Presto 之间构建 PoC 集成,但 Presto 并不真的想与 Spark 交谈,我有点碰壁了。

我正在使用 Presto 中的连接器

hive

connector.name=hive-hadoop2
hive.metastore.uri=thrift://docker-spark-spark-1:10000
#more s3 related props

并在 docker 中启动了 ThriftServer。

ThriftServer 响应良好,从 Beeline 连接时创建数据库等。 从 presto cli 做同样的事情会得到回报

docker-spark-presto-1  | 2023-05-15T10:26:38.453Z   DEBUG   Query-20230515_102637_00001_xt5vr-174   com.facebook.presto.hive.RetryDriver    Failed on executing getTable with attempt 1, will retry. Exception: docker-spark-spark-1:10000: null

最终产生

docker-spark-presto-1  | com.facebook.presto.spi.PrestoException: docker-spark-spark-1:10000: null
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:349)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.BridgingHiveMetastore.getAllTables(BridgingHiveMetastore.java:157)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadAllTables(CachingHiveMetastore.java:493)
docker-spark-presto-1  |    at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
docker-spark-presto-1  |    at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:188)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
docker-spark-presto-1  |    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:326)
....
....
docker-spark-presto-1  | Caused by: org.apache.thrift.transport.TTransportException: docker-spark-spark-1:10000: null
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.Transport.rewriteException(Transport.java:92)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.Transport.access$000(Transport.java:32)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:169)
docker-spark-presto-1  |    at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
docker-spark-presto-1  |    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
docker-spark-presto-1  |    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
docker-spark-presto-1  |    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_tables(ThriftHiveMetastore.java:1966)
docker-spark-presto-1  |    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_tables(ThriftHiveMetastore.java:1953)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastoreClient.getAllTables(ThriftHiveMetastoreClient.java:110)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$null$8(ThriftHiveMetastore.java:325)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getMetastoreClientThenCall(ThriftHiveMetastore.java:1146)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$getAllTables$9(ThriftHiveMetastore.java:325)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:48)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$getAllTables$12(ThriftHiveMetastore.java:337)
docker-spark-presto-1  |    at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:139)
docker-spark-presto-1  |    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:336)
docker-spark-presto-1  |    ... 67 more
docker-spark-presto-1  |    Suppressed: org.apache.thrift.transport.TTransportException: docker-spark-spark-1:10000: null
docker-spark-presto-1  |        ... 83 more 

最让我困扰的是使用 presto-cli 没有日志进入 thriftserver。 有没有人经历过类似的事情或有任何想法?

apache-spark hive presto
© www.soinside.com 2019 - 2024. All rights reserved.