我们在 AWS EKS 基础设施上遇到了一个问题，即使用 Hazelcast (5.0.2) 和 Kubernetes Discovery 插件 (2.2.3) 的应用程序无法将自身检测为成员以及其他类似的 pod - 这是2 个 Pod 部署。根据日志，Hazelcast Kubernetes 插件似乎无法连接到 Kubernetes API。遵循此处找到的说明 - https://github.com/hazelcast/hazelcast-kubernetes。
除了此处提到的内容之外，我无法找到关于与 AWS EKS/EC2 类型基础设施一起使用哪种类型的 Discovery 插件的大量指南： https://docs.hazelcast.com/imdg/4.2/plugins/cloud-discovery#hazelcast-cloud-discovery-plugins-aws
我们曾尝试使用 AWS 插件，但根据该页面，该插件似乎只能与 AWS ECS/EC2 或纯 EC2 实现一起使用。使用该插件确实设法让每个 pod 启动自己的节点，但无法检测到任何其他节点 - 因此我们恢复使用所示的 Kubernetes 插件。
我们迁移到 AWS 的实现在裸机 Kubernetes (v 1.18) 上按预期工作，但在 AWS EKS (Kubernetes v 1.19) 上似乎失败。
我们使用服务名称设置在特定命名空间下进行检测，以及分配给 2 个 pod 的服务帐户，该帐户具有对 api 的完全访问权限，并且可以使用 wget 或curl 从 pod 进行连接以获取 API REST 响应及其证书和令牌。然而，似乎在某些时候发生了某种 SSL 握手错误，并且发现/连接到服务失败：
com.hazelcast.spi.exception.RestClientException：执行 REST 调用失败 原因：javax.net.ssl.SSLHandshakeException：收到致命警报：handshake_failure
服务 yaml 如下所示：
kind: Service
apiVersion: v1
metadata:
name: my-service-name
namespace: my-namespace
spec:
ports:
- protocol: TCP
port: 5701
targetPort: 5701
selector:
app: my-app
type: ClusterIP
启用的附加日志记录没有提供比下面看到的更多的提示。最初，人们认为其他 AWS 设置（IAM 角色/策略和安全组设置）可能正在发挥作用，但是针对不同的应用程序组件使用稍微不同的版本（Hazelcast 4.2.4 和 Kubernetes Discovery 插件 2.2.2）实现了一个单独的系统，在有状态集中按预期工作。
日志显示以下内容：
[ ] 05-May-2022 06:16:59.918 INFO o.s.b.w.e.tomcat.TomcatWebServer.initialize 90 - Tomcat initialized with port(s): 8080 (http)
[ ] 05-May-2022 06:16:59.947 INFO org.apache.juli.logging.DirectJDKLog.log 173 - Starting service [Tomcat]
[ ] 05-May-2022 06:16:59.947 INFO org.apache.juli.logging.DirectJDKLog.log 173 - Starting Servlet Engine: Apache Tomcat/9.0.13
[ ] 05-May-2022 06:16:59.959 INFO org.apache.juli.logging.DirectJDKLog.log 173 - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/opt/jdk/lib/server:/opt/jdk/lib:/opt/jdk/../lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
[ ] 05-May-2022 06:17:00.046 INFO org.apache.juli.logging.DirectJDKLog.log 173 - Initializing Spring embedded WebApplicationContext
[ ] 05-May-2022 06:17:00.046 INFO o.s.b.w.s.c.ServletWebServerApplicationContext.prepareWebApplicationContext 296 - Root WebApplicationContext: initialization completed in 3391 ms
[ ] 05-May-2022 06:17:01.085 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
[ ] 05-May-2022 06:17:01.291 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2]
+ + o o o o---o o----o o o---o o o----o o--o--o
+ + + + | | / \ / | | / / \ | |
+ + + + + o----o o o o o----o | o o o o----o |
+ + + + | | / \ / | | \ / \ | |
+ + o o o o o---o o----o o----o o---o o o o----o o
[ ] 05-May-2022 06:17:01.291 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Copyright (c) 2008-2021, Hazelcast, Inc. All Rights Reserved.
[ ] 05-May-2022 06:17:01.291 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Hazelcast Platform 5.0.2 (20211221 - 18eec9f) starting at [192.168.50.110]:5701
[ ] 05-May-2022 06:17:01.291 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Cluster name: dev
[ ] 05-May-2022 06:17:01.291 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] The Jet engine is disabled.
To enable the Jet engine on the members, please do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true
- Add environment variable: HZ_JET_ENABLED=true
[ ] 05-May-2022 06:17:01.687 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: my-service-name, service-port: 0, service-label: null, service-label-value: true, namespace: my-namespace, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
[ ] 05-May-2022 06:17:01.690 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Kubernetes Discovery activated with mode: KUBERNETES_API
[ ] 05-May-2022 06:17:01.692 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see ?? security recommendations and the status of current config.
[ ] 05-May-2022 06:17:01.764 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Using Discovery SPI
[ ] 05-May-2022 06:17:01.768 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
[ ] 05-May-2022 06:17:02.010 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
[ ] 05-May-2022 06:17:02.016 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] [192.168.50.110]:5701 is STARTING
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/service.jar!/BOOT-INF/lib/hazelcast-5.0.2.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[ ] 05-May-2022 06:17:02.209 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [1] retrying in 1 seconds...
[ ] 05-May-2022 06:17:03.715 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [2] retrying in 2 seconds...
[ ] 05-May-2022 06:17:05.969 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [3] retrying in 3 seconds...
[ ] 05-May-2022 06:17:09.350 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Cannot fetch the current zone, ZONE_AWARE feature is disabled
[ ] 05-May-2022 06:17:09.356 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [1] retrying in 1 seconds...
[ ] 05-May-2022 06:17:10.861 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [2] retrying in 2 seconds...
[ ] 05-May-2022 06:17:13.117 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [3] retrying in 3 seconds...
[ ] 05-May-2022 06:17:16.496 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Cannot fetch name of the node, NODE_AWARE feature is disabled
[ ] 05-May-2022 06:17:16.499 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [1] retrying in 1 seconds...
[ ] 05-May-2022 06:17:18.004 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [2] retrying in 2 seconds...
[ ] 05-May-2022 06:17:20.258 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - Couldn't connect to the service, [3] retrying in 3 seconds...
[ ] 05-May-2022 06:17:23.641 ERROR c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Failure in executing REST call
com.hazelcast.spi.exception.RestClientException: Failure in executing REST call
at com.hazelcast.spi.utils.RestClient.call(RestClient.java:163)
at com.hazelcast.spi.utils.RestClient.lambda$callWithRetries$0(RestClient.java:130)
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65)
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:51)
at com.hazelcast.spi.utils.RestClient.callWithRetries(RestClient.java:130)
at com.hazelcast.spi.utils.RestClient.get(RestClient.java:122)
at com.hazelcast.kubernetes.KubernetesClient.lambda$callGet$4(KubernetesClient.java:557)
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65)
at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:554)
at com.hazelcast.kubernetes.KubernetesClient.endpointsByName(KubernetesClient.java:129)
at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:62)
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:136)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:72)
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:71)
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:60)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:135)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:96)
at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)
at com.hazelcast.instance.impl.Node.join(Node.java:808)
at com.hazelcast.instance.impl.Node.start(Node.java:470)
at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:124)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61)
at at.company.product.config.HazelCastConfiguration.hazelcastInstance(HazelCastConfiguration.java:44)
at at.company.product.config.HazelCastConfiguration$$EnhancerBySpringCGLIB$$8dff12a6.CGLIB$hazelcastInstance$1(<generated>)
at at.company.product.config.HazelCastConfiguration$$EnhancerBySpringCGLIB$$8dff12a6$$FastClassBySpringCGLIB$$62bdb2d8.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
at at.company.product.config.HazelCastConfiguration$$EnhancerBySpringCGLIB$$8dff12a6.hazelcastInstance(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:607)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:307)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at at.company.product.ClassServiceApplication.main(ClassServiceApplication.java:28)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:340)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
at com.hazelcast.spi.utils.RestClient.checkResponseCode(RestClient.java:173)
at com.hazelcast.spi.utils.RestClient.call(RestClient.java:160)
... 65 common frames omitted
[ ] 05-May-2022 06:17:23.642 ERROR c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Could not join cluster. Shutting down now!
[ ] 05-May-2022 06:17:23.642 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] [192.168.50.110]:5701 is SHUTTING_DOWN
[ ] 05-May-2022 06:17:23.645 WARN c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Terminating forcefully...
[ ] 05-May-2022 06:17:23.645 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Shutting down connection manager...
[ ] 05-May-2022 06:17:23.647 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Shutting down node engine...
[ ] 05-May-2022 06:17:23.654 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Destroying node NodeExtension.
[ ] 05-May-2022 06:17:23.655 INFO c.h.l.StandardLoggerFactory$StandardLogger.log 56 - [192.168.50.110]:5701 [dev] [5.0.2] Hazelcast Shutdown is completed in 10 ms.
Hazelcast 5.x 已弃用 hazelcase-kubernetes 插件，并已合并以直接包含在 Hazelcast 中 (https://docs.hazelcast.com/hazelcast/5.0/deploy/deploying-in-kubernetes.html ）。 因此，正如您所发现的，使用 Hazelcast 4.x 的比较应用程序正在使用 Kubernetes 插件。对于 Hazelcast 5.x，您不得包含该插件。
从您提到的配置来看，听起来您正在通过无头服务（通过 DNS）和 Kubernetes API 混合两种发现方法。
请尝试仅采用一种方法 - 我建议在检查您发布的服务 YAML 后使用 DNS 发现。
此外，在 Kubernetes 上运行时，您可能需要根据客户端文档调整并仔细检查 Hazelcast 配置（如果适用）。 您可以发布使用的 Hazelcast 配置（如果它源自文档中的示例）吗？