Apache Flink 通过 Jenkins 和 Spinnaker 提交作业时抛出异常

问题描述 投票:0回答:1

我们正在使用配置为独立 Kubernetes Pod 的 Apache Flink 1.16.1,以便我们的应用程序之一从融合的 Kafka 主题中读取数据以进行事件关联。我们使用 flink 的 Table API join 进行相同的操作(以 SQL 格式)。我们能够使用 flink 的 UI 提交作业。对于我们的 DEV 环境,我们实现了一个 jenkins 管道,它下载提交作业所需的 jar,并创建 flink kubernetes pod,并将下载的 jar 复制到 flink pod 的文件夹,并使用 flink 的 run 命令提交作业。部署步骤通过 spinnaker webhook 进行。我们使用 docker 文件来创建 kubernetes pod,还有一个 docker-entrypoin.sh,其中包含用于提交作业的 flink run 命令。

一切正常,但是当提交作业时,我们收到以下异常。

使用的flink运行命令是

flink run  /opt/flink/lib/application-0.0.1.jar

如有任何帮助,我们将不胜感激。

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute sql
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:843)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165)
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165)
Caused by: org.apache.flink.table.api.TableException: Failed to execute sql
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:867)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730)
        at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
        ... 8 more
Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'insert-into_default_catalog.default_database.Sample'.
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2203)
        at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:206)
        at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:95)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:850)
        ... 17 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
15:06:44.964 [main] ERROR org.apache.flink.client.cli.CliFrontend - Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute sql
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:843) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087) ~[application-0.0.1.jar:?]
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165) ~[application-0.0.1.jar:?]
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [application-0.0.1.jar:?]
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165) [application-0.0.1.jar:?]
aused by: org.apache.flink.table.api.TableException: Failed to execute sql
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:867) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730) ~[application-0.0.1.jar:?]
        at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157) ~[application-0.0.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[application-0.0.1.jar:?]
        ... 8 more
Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'insert-into_default_catalog.default_database.Sample'.
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2203) ~[flink-dist-1.16.1.jar:1.16.1]
        at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:206) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:95) ~[?:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:850) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918) ~[application-0.0.1.jar:?]
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730) ~[application-0.0.1.jar:?]
        at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157) ~[application-0.0.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[application-0.0.1.jar:?]
        ... 8 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
        at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$11(RestClusterClient.java:448) ~[application-0.0.1.jar:?]
        at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
        at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:271) ~[application-0.0.1.jar:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
        at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125) ~[application-0.0.1.jar:?]
        at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217) ~[application-0.0.1.jar:?]
        at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489) ~[application-0.0.1.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.TimeoutException
        at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125) ~[application-0.0.1.jar:?]
        at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217) ~[application-0.0.1.jar:?]
        at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489) ~[application-0.0.1.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
        at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$11(RestClusterClient.java:448)
        at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:271)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125)
        at org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
        at org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException
        ... 9 more
apache-flink flink-streaming flink-sql
1个回答
0
投票

您找到解决问题的方法了吗?目前遇到同样的问题。

© www.soinside.com 2019 - 2024. All rights reserved.