我正在 Minio、Spark 和 Presto 之间构建 PoC 集成,但 Presto 并不真的想与 Spark 交谈,我有点碰壁了。
我正在使用 Presto 中的连接器
hive
:
connector.name=hive-hadoop2
hive.metastore.uri=thrift://docker-spark-spark-1:10000
#more s3 related props
并在 docker 中启动了 ThriftServer。
ThriftServer 响应良好,从 Beeline 连接时创建数据库等。 从 presto cli 做同样的事情会得到回报
docker-spark-presto-1 | 2023-05-15T10:26:38.453Z DEBUG Query-20230515_102637_00001_xt5vr-174 com.facebook.presto.hive.RetryDriver Failed on executing getTable with attempt 1, will retry. Exception: docker-spark-spark-1:10000: null
最终产生
docker-spark-presto-1 | com.facebook.presto.spi.PrestoException: docker-spark-spark-1:10000: null
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:349)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.BridgingHiveMetastore.getAllTables(BridgingHiveMetastore.java:157)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadAllTables(CachingHiveMetastore.java:493)
docker-spark-presto-1 | at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
docker-spark-presto-1 | at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:188)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
docker-spark-presto-1 | at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
docker-spark-presto-1 | at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
docker-spark-presto-1 | at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.CachingHiveMetastore.get(CachingHiveMetastore.java:326)
....
....
docker-spark-presto-1 | Caused by: org.apache.thrift.transport.TTransportException: docker-spark-spark-1:10000: null
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.Transport.rewriteException(Transport.java:92)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.Transport.access$000(Transport.java:32)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:169)
docker-spark-presto-1 | at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
docker-spark-presto-1 | at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
docker-spark-presto-1 | at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
docker-spark-presto-1 | at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_tables(ThriftHiveMetastore.java:1966)
docker-spark-presto-1 | at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_tables(ThriftHiveMetastore.java:1953)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastoreClient.getAllTables(ThriftHiveMetastoreClient.java:110)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$null$8(ThriftHiveMetastore.java:325)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getMetastoreClientThenCall(ThriftHiveMetastore.java:1146)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$getAllTables$9(ThriftHiveMetastore.java:325)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:48)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$getAllTables$12(ThriftHiveMetastore.java:337)
docker-spark-presto-1 | at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:139)
docker-spark-presto-1 | at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getAllTables(ThriftHiveMetastore.java:336)
docker-spark-presto-1 | ... 67 more
docker-spark-presto-1 | Suppressed: org.apache.thrift.transport.TTransportException: docker-spark-spark-1:10000: null
docker-spark-presto-1 | ... 83 more
最让我困扰的是使用 presto-cli 没有日志进入 thriftserver。 有没有人经历过类似的事情或有任何想法?