Hazelcast 集群无法连接到 Kubernetes Master

问题描述 投票:0回答:1

AKS 升级后,我重新部署了一个已经运行的应用程序。 现在我的 Pod 无法相互通信,表示无法连接到 Kubernetes Master 。我尝试在我的服务和部署中外部化端口 5701。 造成这种情况的可能原因是什么?

2023-07-25 11:40:15.703 | forecast-management-service | main | ERROR | w.p.i.c.IAMCCMControlsConfigBaseListener |  | Unable to find SCM configuration for IAMEnvControlsConfig, default value for properties will be used and no property will be read from CCM
2023-07-25 11:40:23.593 | forecast-management-service | main | INFO  | com.hazelcast.instance.AddressPicker     |  | [LOCAL] [dev] [3.11.2] Prefer IPv4 stack is true, prefer IPv6 addresses is false
2023-07-25 11:40:23.601 | forecast-management-service | main | INFO  | com.hazelcast.instance.AddressPicker     |  | [LOCAL] [dev] [3.11.2] Picked [10.112.5.51]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2023-07-25 11:40:23.642 | forecast-management-service | main | INFO  | com.hazelcast.system                     |  | [10.112.5.51]:5701 [dev] [3.11.2] Hazelcast 3.11.2 (20190216 - e302506) starting at [10.112.5.51]:5701
2023-07-25 11:40:23.643 | forecast-management-service | main | INFO  | com.hazelcast.system                     |  | [10.112.5.51]:5701 [dev] [3.11.2] Copyright (c) 2008-2019, Hazelcast, Inc. All Rights Reserved.
2023-07-25 11:40:23.643 | forecast-management-service | main | INFO  | com.hazelcast.instance.Node              |  | [10.112.5.51]:5701 [dev] [3.11.2] A non-empty group password is configured for the Hazelcast member. Starting with Hazelcast version 3.8.2, members with the same group name, but with different group passwords (that do not use authentication) form a cluster. The group password configuration will be removed completely in a future release.
2023-07-25 11:40:24.218 | forecast-management-service | main | INFO  | c.h.s.i.o.impl.BackpressureRegulator     |  | [10.112.5.51]:5701 [dev] [3.11.2] Backpressure is disabled
2023-07-25 11:40:25.203 | forecast-management-service | main | INFO  | c.h.s.d.integration.DiscoveryService     |  | [10.112.5.51]:5701 [dev] [3.11.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: forecast-mgmt, service-port: 5701, service-label: null, service-label-value: true, namespace: qa-forecast-mgmt, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
2023-07-25 11:40:25.210 | forecast-management-service | main | INFO  | c.h.s.d.integration.DiscoveryService     |  | [10.112.5.51]:5701 [dev] [3.11.2] Kubernetes Discovery activated resolver: KubernetesApiEndpointResolver
2023-07-25 11:40:25.341 | forecast-management-service | main | INFO  | com.hazelcast.instance.Node              |  | [10.112.5.51]:5701 [dev] [3.11.2] Activating Discovery SPI Joiner
2023-07-25 11:40:25.693 | forecast-management-service | main | INFO  | c.h.s.i.o.impl.OperationExecutorImpl     |  | [10.112.5.51]:5701 [dev] [3.11.2] Starting 7 partition threads and 4 generic threads (1 dedicated for priority tasks)
2023-07-25 11:40:25.703 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.Diagnostics     |  | [10.112.5.51]:5701 [dev] [3.11.2] Diagnostics started
2023-07-25 11:40:25.704 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.BuildInfoPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 11:40:25.704 | forecast-management-service | main | INFO  | c.h.i.diagnostics.SystemPropertiesPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 11:40:25.705 | forecast-management-service | main | INFO  | c.h.i.diagnostics.ConfigPropertiesPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 11:40:25.709 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.MetricsPlugin   |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 11:40:25.711 | forecast-management-service | main | INFO  | c.h.i.diagnostics.SlowOperationPlugin    |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 11:40:25.712 | forecast-management-service | main | INFO  | .h.i.d.MemberHazelcastInstanceInfoPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 11:40:25.713 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.SystemLogPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active: logPartitions:false
2023-07-25 11:40:25.718 | forecast-management-service | main | INFO  | .h.i.d.MemberHazelcastInstanceInfoPlugin |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active, period-millis:10000
2023-07-25 11:40:25.718 | forecast-management-service | main | INFO  | c.h.i.d.OperationHeartbeatPlugin         |  | [10.112.5.51]:5701 [dev] [3.11.2] Plugin:active: period-millis:10000 max-deviation:33%
2023-07-25 11:40:25.722 | forecast-management-service | main | INFO  | com.hazelcast.core.LifecycleService      |  | [10.112.5.51]:5701 [dev] [3.11.2] [10.112.5.51]:5701 is STARTING
2023-07-25 11:40:25.756 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...
2023-07-25 11:40:27.262 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [2] retrying in 2 seconds...
2023-07-25 11:40:29.519 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [3] retrying in 3 seconds...
2023-07-25 11:40:32.899 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [4] retrying in 5 seconds...
2023-07-25 11:40:32.995 | forecast-management-service | hz.jobEngine-hazelcast-kubernetes.IO.thread-in-0 | INFO  | com.hazelcast.nio.tcp.TcpIpConnection    |  | [10.112.5.51]:5701 [dev] [3.11.2] Connection[id=1, /10.112.5.51:5701->/10.112.4.237:44935, endpoint=null, alive=false, type=NONE] closed. Reason: Connection closed by the other side
2023-07-25 11:40:37.967 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [5] retrying in 7 seconds...
2023-07-25 11:40:43.236 | forecast-management-service | hz.jobEngine-hazelcast-kubernetes.IO.thread-in-1 | INFO  | com.hazelcast.nio.tcp.TcpIpConnection    |  | [10.112.5.51]:5701 [dev] [3.11.2] Connection[id=2, /10.112.5.51:5701->/10.112.4.208:32261, endpoint=null, alive=false, type=NONE] closed. Reason: Connection closed by the other side

编辑1:

删除端口外部化标签后,下面添加了默认 Kubernetes 行为的日志。它说一些与 Zone Aware 相关的警告。 最近我们更改了项目的 Git 存储库,但

ServiceName
Namespace
已正确映射。

2023-07-25 13:20:44.153 | forecast-management-service | main | INFO  | c.h.s.d.integration.DiscoveryService     |  | [10.112.5.131]:5701 [dev] [3.11.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: forecast-mgmt, service-port: 5701, service-label: null, service-label-value: true, namespace: qa-forecast-mgmt, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
2023-07-25 13:20:44.160 | forecast-management-service | main | INFO  | c.h.s.d.integration.DiscoveryService     |  | [10.112.5.131]:5701 [dev] [3.11.2] Kubernetes Discovery activated resolver: KubernetesApiEndpointResolver
2023-07-25 13:20:44.281 | forecast-management-service | main | INFO  | com.hazelcast.instance.Node              |  | [10.112.5.131]:5701 [dev] [3.11.2] Activating Discovery SPI Joiner
2023-07-25 13:20:44.618 | forecast-management-service | main | INFO  | c.h.s.i.o.impl.OperationExecutorImpl     |  | [10.112.5.131]:5701 [dev] [3.11.2] Starting 7 partition threads and 4 generic threads (1 dedicated for priority tasks)
2023-07-25 13:20:44.629 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.Diagnostics     |  | [10.112.5.131]:5701 [dev] [3.11.2] Diagnostics started
2023-07-25 13:20:44.630 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.BuildInfoPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 13:20:44.631 | forecast-management-service | main | INFO  | c.h.i.diagnostics.SystemPropertiesPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 13:20:44.631 | forecast-management-service | main | INFO  | c.h.i.diagnostics.ConfigPropertiesPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active
2023-07-25 13:20:44.634 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.MetricsPlugin   |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 13:20:44.636 | forecast-management-service | main | INFO  | c.h.i.diagnostics.SlowOperationPlugin    |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 13:20:44.636 | forecast-management-service | main | INFO  | .h.i.d.MemberHazelcastInstanceInfoPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active, period-millis:60000
2023-07-25 13:20:44.638 | forecast-management-service | main | INFO  | c.h.internal.diagnostics.SystemLogPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active: logPartitions:false
2023-07-25 13:20:44.643 | forecast-management-service | main | INFO  | .h.i.d.MemberHazelcastInstanceInfoPlugin |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active, period-millis:10000
2023-07-25 13:20:44.643 | forecast-management-service | main | INFO  | c.h.i.d.OperationHeartbeatPlugin         |  | [10.112.5.131]:5701 [dev] [3.11.2] Plugin:active: period-millis:10000 max-deviation:33%
2023-07-25 13:20:44.646 | forecast-management-service | main | INFO  | com.hazelcast.core.LifecycleService      |  | [10.112.5.131]:5701 [dev] [3.11.2] [10.112.5.131]:5701 is STARTING
2023-07-25 13:20:44.679 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...
2023-07-25 13:20:46.184 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [2] retrying in 2 seconds...
2023-07-25 13:20:48.443 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [3] retrying in 3 seconds...
2023-07-25 13:20:51.823 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [4] retrying in 5 seconds...
2023-07-25 13:20:56.893 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [5] retrying in 7 seconds...
2023-07-25 13:21:04.492 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [6] retrying in 11 seconds...
2023-07-25 13:21:15.890 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [7] retrying in 17 seconds...
2023-07-25 13:21:32.978 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [8] retrying in 25 seconds...
2023-07-25 13:21:58.610 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [9] retrying in 38 seconds...
2023-07-25 13:22:37.056 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [10] retrying in 57 seconds...
2023-07-25 13:23:34.725 | forecast-management-service | main | WARN  | c.h.s.d.integration.DiscoveryService     |  | [10.112.5.131]:5701 [dev] [3.11.2] Cannot fetch the current zone, ZONE_AWARE feature is disabled
2023-07-25 13:23:34.733 | forecast-management-service | main | WARN  | com.hazelcast.kubernetes.RetryUtils      |  | Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...```
java spring-boot kubernetes hazelcast
1个回答
0
投票

假设您没有网络连接问题,因此正如警告所示,Hazelcast client 无法连接到 Kubernetes 主 API 服务器,这可能是由于 Hazelcast 客户端或 Kubernetes 集群配置错误造成的。 这里有一些解决方法,

  • 尝试重新启动 Hazelcast 客户端和 Kubernetes API 服务器 pod
    kubectl delete pod 'the-pod-name' -n 'the-namespace'
    ,这会在创建和启动新 pod 时导致服务短暂中断。但是,如果您有多个正在运行的 API 服务器 Pod 副本,则可以使用滚动更新策略来更新 Pod。(根据我的经验,这主要发生)
  • 确保在 Hazelcast 客户端配置中正确设置
    ServiceName
    Namespace
    属性。
  • 检查 Kubernetes 配置文件或可能的网络连接问题。
© www.soinside.com 2019 - 2024. All rights reserved.