我有一个运行 Spark/Iceberg/MinIO 的应用程序以及 Iceberg 的 Nessie 目录,一切正常(Iceberg 数据已正确加载到 MinIO 和 Nessie)。
现在我尝试连接 Trino,但对“http://localhost:19120/api/v1/trees/tree/main?fetch=MINIMAL”执行 GET 请求时失败(当我从浏览器执行时,此请求有效,或者失眠)。
在 Trino GUI 中,这是由“java.net.ConnectException:未定义”引起的。
我错过了什么?
我正在使用 Trino 官方 Helm Chart 0.18.0(运行 Trino 版本 432)并在 Nessie 0.77.1(当前最新)和 0.71.0(与 Trino 432 pom.xml 中的 Nessie 版本匹配:https:/ /github.com/trinodb/trino/blob/432/plugin/trino-iceberg/pom.xml
我的 Trino 连接器属性:
connector.name=iceberg
iceberg.catalog.type=nessie
iceberg.nessie-catalog.uri=http://localhost:19120/api/v1
iceberg.nessie-catalog.default-warehouse-dir=s3a://data/test
hive.s3.aws-access-key=VUtRVIf0hg7szCp3k0Pz
hive.s3.aws-secret-key=lHzYClEjh2AH5mEfRdPS720pMl3UZl7riR3uL4pL
hive.s3.endpoint=http://127.0.0.1:9000
hive.s3.path-style-access=true
hive.s3.ssl.enabled=false
我的 Spark 会话配置(Java):
SparkSession.builder()
.appName("Test App")
.master("local")
.config("spark.ui.enabled", "false")
.config("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.config("fs.s3a.endpoint", "http://127.0.0.1:9000")
.config("fs.s3a.access.key", VUtRVIf0hg7szCp3k0Pz)
.config("fs.s3a.secret.key", lHzYClEjh2AH5mEfRdPS720pMl3UZl7riR3uL4pL)
.config("fs.s3a.path.style.access", "true")
.config("spark.sql.warehouse.dir", WAREHOUSE_PATH)
.config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.3.0,org.projectnessie.nessie-integrations:nessie-spark-extensions-3.5_2.13:0.77.1")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions")
.config("spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.nessie.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog")
.config("spark.sql.catalog.nessie.authentication.type", "NONE")
.config("spark.sql.catalog.nessie.uri", "http://localhost:19120/api/v1")
.config("spark.sql.catalog.nessie.ref", "main")
.config("spark.sql.defaultCatalog", "nessie")
.config("spark.sql.catalog.nessie.warehouse", "s3a://data/test")
.getOrCreate();
来自 Trino GUI 的整个堆栈跟踪:
io.trino.spi.TrinoException: Error listing schemas for catalog nessie: Failed to execute GET request against 'http://localhost:19120/api/v1/trees/tree/main?fetch=MINIMAL'.
at io.trino.metadata.MetadataListing.handleListingException(MetadataListing.java:349)
at io.trino.metadata.MetadataListing.listSchemas(MetadataListing.java:97)
at io.trino.metadata.MetadataListing.listSchemas(MetadataListing.java:88)
at io.trino.connector.informationschema.InformationSchemaPageSource.addSchemataRecords(InformationSchemaPageSource.java:322)
at io.trino.connector.informationschema.InformationSchemaPageSource.buildPages(InformationSchemaPageSource.java:226)
at io.trino.connector.informationschema.InformationSchemaPageSource.getNextPage(InformationSchemaPageSource.java:184)
at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:266)
at io.trino.operator.Driver.processInternal(Driver.java:395)
at io.trino.operator.Driver.lambda$process$8(Driver.java:298)
at io.trino.operator.Driver.tryWithLock(Driver.java:694)
at io.trino.operator.Driver.process(Driver.java:290)
at io.trino.operator.Driver.processForDuration(Driver.java:261)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:887)
at io.trino.execution.executor.timesharing.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:187)
at io.trino.execution.executor.timesharing.TimeSharingTaskExecutor$TaskRunner.run(TimeSharingTaskExecutor.java:565)
at io.trino.$gen.Trino_432____20240219_102654_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.projectnessie.client.http.HttpClientException: Failed to execute GET request against 'http://localhost:19120/api/v1/trees/tree/main?fetch=MINIMAL'.
at org.projectnessie.client.http.impl.jdk11.JavaRequest.executeRequest(JavaRequest.java:128)
at org.projectnessie.client.http.HttpRequest.get(HttpRequest.java:80)
at org.projectnessie.client.rest.v1.RestV1TreeClient.getReferenceByName(RestV1TreeClient.java:83)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.projectnessie.client.rest.v1.RestV1Client$ExceptionRewriter.invoke(RestV1Client.java:78)
at jdk.proxy4/jdk.proxy4.$Proxy236.getReferenceByName(Unknown Source)
at org.projectnessie.client.rest.v1.HttpGetReference.get(HttpGetReference.java:34)
at org.apache.iceberg.nessie.NessieIcebergClient.loadReference(NessieIcebergClient.java:114)
at org.apache.iceberg.nessie.NessieIcebergClient.lambda$new$0(NessieIcebergClient.java:81)
at org.apache.iceberg.relocated.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:181)
at org.apache.iceberg.nessie.NessieIcebergClient.getRef(NessieIcebergClient.java:89)
at org.apache.iceberg.nessie.NessieIcebergClient.withReference(NessieIcebergClient.java:501)
at org.apache.iceberg.nessie.NessieIcebergClient.listNamespaces(NessieIcebergClient.java:207)
at io.trino.plugin.iceberg.catalog.nessie.TrinoNessieCatalog.listNamespaces(TrinoNessieCatalog.java:104)
at io.trino.plugin.iceberg.IcebergMetadata.listSchemaNames(IcebergMetadata.java:355)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.listSchemaNames(ClassLoaderSafeConnectorMetadata.java:197)
at io.trino.tracing.TracingConnectorMetadata.listSchemaNames(TracingConnectorMetadata.java:131)
at io.trino.metadata.MetadataManager.listSchemaNames(MetadataManager.java:254)
at io.trino.tracing.TracingMetadata.listSchemaNames(TracingMetadata.java:168)
at io.trino.metadata.MetadataListing.doListSchemas(MetadataListing.java:103)
at io.trino.metadata.MetadataListing.listSchemas(MetadataListing.java:94)
... 17 more
Caused by: java.net.ConnectException: undefined
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:951)
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
at org.projectnessie.client.http.impl.jdk11.JavaHttpClient.lambda$newRequest$0(JavaHttpClient.java:68)
at org.projectnessie.client.http.impl.jdk11.JavaRequest.executeRequest(JavaRequest.java:110)
... 38 more
Caused by: java.net.ConnectException: undefined
at java.net.http/jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1028)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:227)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:280)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$2(PlainHttpConnection.java:238)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
... 3 more
Caused by: java.nio.channels.ClosedChannelException: undefined
at java.base/sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:202)
at java.base/sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:786)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:874)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:210)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:212)
... 9 more
确保可以从 Trino 服务器内访问 Nessie 服务器。我有同样的错误并意识到我设置了
iceberg.nessie-catalog.uri=http://localhost:19120/api/v1
但是,我在 Docker 容器内运行 Trino,因此 localhost 引用了自身。设置
--network=host
解决了这个问题,但您也可以在同一网络下配置两个容器(如果您也在容器中运行 Nessie)。