我想在 docker 环境中使用 SASL_SSL 设置 Kafka kafka 应该可以接收通过公共互联网加密的消息 此外，后端使用了 telegraf grafana 等，一切都运行完美
只允许zookeeper和kafka在互联网上通信
kafka 应使用 SASL_SSL SCRAM-SHA-512 加密
kafka和zookeeper之间的连接应该通过DIGEST-MD5运行
但我找不到解决方案，因为我总是收到以下错误
错误使用登录上下文“客户端”SASL 身份验证失败。 （org.apache.zookeeper.client.ZooKeeperSaslClient
** ERROR 连接到 Zookeeper 服务器时发生错误[zookeeper:2181,zookeeper:2182]。认证失败。 **
** javax.security.sasl.SaslException：与 Zookeeper 仲裁成员进行身份验证时出错：仲裁成员的 saslToken 为 null。**
我还添加了完整的日志
===> Configuring ...
Running in Zookeeper mode...
SSL is enabled.
SASL is enabled.
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Check if Zookeeper is healthy ...
[2023-08-14 14:15:47,648] INFO SASL is enabled. java.security.auth.login.config=/etc/kafka/sasl.jaas.config (io.confluent.admin.utils.ClusterStatus)
[2023-08-14 14:15:47,677] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:host.name=353d245d9a35 (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:java.version=11.0.18 (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,678] INFO Client environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.ZooKeeper)
8-14 14:15:47,678] INFO Client environment:java.class.path=/usr/share/java/cp-base-new/zookeeper-jute-3.6.3.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.14.2.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/kafka-storage-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-databind-2.14.2.jar:/usr/share/java/cp-base-new/jackson-annotations-2.14.2.jar:/usr/share/java/cp-base-new/disk-usage-agent-7.4.1.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.10.jar:/usr/share/java/cp-base-new/kafka-metadata-7.4.1-ccs.jar:/usr/share/java/cp-base-new/lz4-java-1.8.0.jar:/usr/share/java/cp-base-new/json-simple-1.1.1.jar:/usr/share/java/cp-base-new/re2j-1.6.jar:/usr/share/java/cp-base-new/snakeyaml-2.0.jar:/usr/share/java/cp-base-new/metrics-core-4.1.12.1.jar:/usr/share/java/cp-base-new/gson-2.9.0.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.36.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.10.0.jar:/usr/share/java/cp-base-new/kafka-group-coordinator-7.4.1-ccs.jar:/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/slf4j-reload4j-1.7.36.jar:/usr/share/java/cp-base-new/zstd-jni-1.5.2-1.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.14.2.jar:/usr/share/java/cp-base-new/jose4j-0.9.3.jar:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar:/usr/share/java/cp-base-new/common-utils-7.4.1.jar:/usr/share/java/cp-base-new/kafka_2.13-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-clients-7.4.1-ccs.jar:/usr/share/java/cp-base-new/snappy-java-1.1.10.1.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/zookeeper-3.6.3.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.4.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-1.0.2.jar:/usr/share/java/cp-base-new/jackson-core-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-jvm-1.7.1.jar:/usr/share/java/cp-base-new/logredactor-1.0.12.jar:/usr/share/java/cp-base-new/kafka-server-common-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-storage-api-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.14.2.jar:/usr/share/java/cp-base-new/kafka-raft-7.4.1-ccs.jar:/usr/share/java/cp-base-new/scala-library-2.13.10.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-core-1.7.1.jar:/usr/share/java/cp-base-new/utility-belt-7.4.1.jar:/usr/share/java/cp-base-new/logredactor-metrics-1.0.12.jar:/usr/share/java/cp-base-new/reload4j-1.2.19.jar:/usr/share/java/cp-base-new/minimal-json-0.9.5.jar (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,678] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:os.version=5.10.0-21-amd64 (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.name=appuser (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.home=/home/appuser (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.dir=/home/appuser (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.free=55MB (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.max=984MB (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.total=62MB (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,687] INFO Initiating client connection, connectString=zookeeper:2181,zookeeper:2182 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@221af3c0 (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,694] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-14 14:15:47,704] INFO jute.maxbuffer value is 1048575 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2023-08-14 14:15:47,715] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,837] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2023-08-14 14:15:47,842] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2023-08-14 14:15:47,884] INFO Opening socket connection to server zookeeper/192.168.112.2:2182. (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,884] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,899] INFO Socket connection established, initiating session, client: /192.168.112.3:47736, server: zookeeper/192.168.112.2:2182 (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,905] WARN Session 0x0 for sever zookeeper/192.168.112.2:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-14 14:15:48,808] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2023-08-14 14:15:48,809] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2023-08-14 14:15:48,810] INFO Opening socket connection to server zookeeper/192.168.112.2:2181. (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,811] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,815] INFO Socket connection established, initiating session, client: /192.168.112.3:58200, server: zookeeper/192.168.112.2:2181 (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,823] INFO Session establishment complete on server zookeeper/192.168.112.2:2181, session id = 0x101527ddfc400fd, negotiated timeout = 40000 (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,842] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:310)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:270)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:936)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-14 14:15:48,843] ERROR Error occurred while connecting to Zookeeper server[zookeeper:2181,zookeeper:2182]. Authentication failed. (io.confluent.admin.utils.ClusterStatus)
[2023-08-14 14:15:48,843] INFO EventThread shut down for session: 0x101527ddfc400fd (org.apache.zookeeper.ClientCnxn)
Using log4j config /etc/kafka/log4j.properties
这是我的配置
服务器属性
# Broker Basics
broker.id=1
listeners=SASL_PLAINTEXT://kafka:9092,SSL://kafka:9093,SASL_SSL://kafka:9094
num.network.threads=3
num.io.threads=8
zookeeper.connect=kafka:2181,kafka:2182
zookeeper.set.acl=true
authorizer.class.name=kafka.security.auth.AclAuthorizer
auto.create.topics.enable=false
num.partitions=1
default.replication.factor=1
socket.request.max.bytes=1000000000
max.request.size=1000000000
security.inter.broker.protocol=SSL
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512,DIGEST-MD5
##CA##
ssl.truststore.location=/etc/kafka/truststore/kafka.truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/etc/kafka/keystore/kafka.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
sasl.enabled.mechanisms=PLAIN
卡夫卡·贾斯
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="secret";
org.apache.zookeeper.server.auth.DigestLoginModule required
username="admin"
password="secret";
};
Client {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="secret";
};
顺便说一句，尝试了妈妈的组合，例如 Server{ } 作为名称等等
zookeeper.properties
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/log
clientPort=2181
tickTime=2000
maxClientCnxns=100
autopurge.snapRetainCount=12
autopurge.purgeInterval=168
initLimit=10
syncLimit=5
secureClientPort=2182
ssl.keyStore.location=/etc/kafka/keystore/zookeeper.keystore.jks
ssl.keyStore.password=password
ssl.trustStore.location=/etc/kafka/truststore/zookeeper.truststore.jks
ssl.trustStore.password=password
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
admin.enableServer=false
Zookeeper.jaas
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_admin="secret";
};
这就是我的 docker Compose：
version: '3.6'
services:
zookeeper:
image: 'confluentinc/cp-zookeeper:latest'
container_name: zookeeper
restart: always
ports:
- '2181:2181'
- '2182:2182'
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SECURE_CLIENT_PORT: 2182
serverCnxnFactory: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_OPTS: -Djava.security.auth.login.config=/etc/kafka/zookeeper.jaas
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 10
ZOOKEEPER_SYNC_LIMIT: 5
ZOOKEEPER_DATADIR_AUTOCREATE: "false"
ZOOKEEPER_MAX_CLIENT_CNXNS: 100
ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT: 12
ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: 168
ZOOKEEPER_ADMIN_ENABLE_SERVER: "false"
ZOOKEEPER_AUTH_PROVIDER: org.apache.zookeeper.server.auth.SASLAuthenticationProvider
ZOOKEEPER_REQUIRE_CLIENT_AUTH_SCHEME: sasl
ZOOKEEPER_JAAS_LOGIN_RENEW: 3600000
ZOOKEEPER_AUTH_PROVIDER_X509: org.apache.zookeeper.server.auth.X509AuthenticationProvider
ZOOKEEPER_SERVER_CNXN_FACTORY: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_SSL_PROTOCOL: TLSv1.2
ZOOKEEPER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/truststore/zookeeper.truststore.jks
ZOOKEEPER_SSL_KEYSTORE_LOCATION: /etc/kafka/keystore/zookeeper.keystore.jks
ZOOKEEPER_SSL_KEYSTORE_PASSWORD: password
ZOOKEEPER_SSL_TRUSTSTORE_PASSWORD: password
ZOOKEEPER_DIGEST_AUTHENTICATION_PROVIDER_SUPERDIGEST: admin:sha1hashpassword
KAFKA_LOG4J_ROOTLOGLEVEL: DEBUG
volumes:
- /data/zookeeper/zookeeper.properties:/etc/kafka/zookeeper.properties
- /data/zookeeper/zookeeper.jaas:/etc/kafka/zookeeper.jaas
- /data/zookeeper/truststore:/etc/kafka/truststore
- /data/zookeeper/keystore:/etc/kafka/keystore
networks:
- kafka_network
kafka:
image: 'confluentinc/cp-kafka:latest'
container_name: kafka
restart: always
ports:
- '9093:9093'
- '9094:9094'
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181,zookeeper:2182
KAFKA_LISTENERS: SSL://kafka:9093,SASL_SSL://kakfa:9094
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,SASL_SSL://kafka:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SSL_KEYSTORE_FILENAME: kafka.keystore.jks
KAFKA_SSL_KEY_CREDENTIALS: password.key
KAFKA_SSL_KEYSTORE_CREDENTIALS: password.key
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/sasl.jaas.config"
volumes:
- /data/kafka/server.properties:/etc/kafka/server.properties
- /data/kafka/keystore/kafka.keystore.jks:/etc/kafka/secrets/kafka.keystore.jks
- /data/kafka/truststore/kafka.truststore.jks:/etc/kafka/secrets/truststore.keystore.jks
- /data/kafka/password.key:/etc/kafka/secrets/password.key
- /data/kafka/sasl.jaas.config/etc/kafka/sasl.jaas.config
networks:
- kafka_network
networks:
kafka_network:
如果您需要更多详细信息，请随时询问 我可以提供更多
如果有人知道在 docker 中构建这个的良好文档或者如果有人知道我做错了什么，这会对我有很大帮助
我现在自己解决了我的问题。
如果你也有同样的问题
强烈建议切换到 Apache KRaft 而不是 Zookeeper，因为 Zookeeper 已经过时了。
也不要使用自签名证书，这可以解决问题。 使用让加密证书或为用例购买证书
还要确保服务器可以使用 sasl 密码读取数据