apache zeppelin 多节点集群

问题描述 投票:0回答:0

我部署了一个有 2 个节点的 zeepline 集群。配置“zeppelin.cluster.addr”,notebook和interpreter存放在hdfs中,使用nginx做负载均衡。在使用中发现任务执行失败。日志中显示raft异常,metadata之间无法同步。 如何正确配置 zeppelin 集群。谢谢!

相关日志(zeppelin-idox-eb6126.log):

ERROR [2023-03-21 18:07:35,496] ({Thread-11} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!
WARN [2023-03-21 18:07:35,496] ({Thread-11} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.
WARN [2023-03-21 18:07:36,106] ({Thread-13} ClusterManager.java[run]:266) - Raft incomplete initialization! retry[6210]
WARN [2023-03-21 18:07:36,661] ({raft-server-10.1.106.26:6000} DelegatingLogger.java[warn]:230) - RaftServer{10.1.106.26:6000}{role=FOLLOWER} - io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /10.1.106.27:6000
ERROR [2023-03-21 18:07:38,497] ({Thread-11} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!
WARN [2023-03-21 18:07:38,497] ({Thread-11} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.
WARN [2023-03-21 18:07:38,759] ({raft-server-10.1.106.26:6000} DelegatingLogger.java[warn]:230) - RaftServer{10.1.106.26:6000}{role=FOLLOWER} - io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /10.1.106.27:6000
WARN [2023-03-21 18:07:39,111] ({Thread-13} ClusterManager.java[run]:266) - Raft incomplete initialization! retry[6240]
WARN [2023-03-21 18:07:40,627] ({raft-server-10.1.106.26:6000} DelegatingLogger.java[warn]:230) - RaftServer{10.1.106.26:6000}{role=FOLLOWER} - io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /10.1.106.27:6000
ERROR [2023-03-21 18:07:41,498] ({Thread-11} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!
WARN [2023-03-21 18:07:41,498] ({Thread-11} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.
WARN [2023-03-21 18:07:42,115] ({Thread-13} ClusterManager.java[run]:266) - Raft incomplete initialization! retry[6270]
WARN [2023-03-21 18:07:42,129] ({raft-server-10.1.106.26:6000} DelegatingLogger.java[warn]:230) - RaftServer{10.1.106.26:6000}{role=FOLLOWER} - io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /10.1.106.27:6000

相关日志(zeppelin-idox-eb6126.log):

org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Creating process hdfs2-shared_process failed on remote server 10.1.61.27:6000
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:129)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:271)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:438)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:69)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:182)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Creating process hdfs2-shared_process failed on remote server 10.1.106.27:6000
at org.apache.zeppelin.interpreter.launcher.ClusterInterpreterLauncher.launchDirectly(ClusterInterpreterLauncher.java:177)
at org.apache.zeppelin.interpreter.launcher.InterpreterLauncher.launch(InterpreterLauncher.java:110)
at org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:856)
at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:66)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:104)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
... 13 more
apache-zeppelin
© www.soinside.com 2019 - 2024. All rights reserved.