Broker 可能无法在 docker compose 中将 kafka 连接器与 kafka 代理连接起来

Question

我正在使用docker容器构建zookeeper、kafka并连接。

version: '2.1'
services:


  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: wurstmeister/kafka
    container_name: kafka-multibinder-1
    ports:
      - "9092:9092"
      - "9094:9094"

    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

      - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
      - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE

    depends_on:
      - zookeeper

  kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9092
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

然而，日志让我感到温暖，“无法成立”。经纪人可能不可用'

kafka-connect_1  | 2020-10-18 04:11:44,671 INFO   ||  Kafka version: 2.5.0   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,672 INFO   ||  Kafka commitId: 66563e712b0b9f84   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,674 INFO   ||  Kafka startTimeMs: 1602994304669   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,806 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:44,918 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:45,024 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:12:44,713 INFO   ||  [AdminClient clientId=adminclient-1] Metadata update failed   [org.apache.kafka.clients.admin.internals.AdminMetadataManager]
kafka-connect_1  | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602994364719) timed out at 9223372036854775807 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
kafka-connect_1  | 2020-10-18 04:12:44,722 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
kafka-connect_1  | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
kafka-connect_1  | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
kafka-connect_1  |  ... 3 more
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Answer 1

在 Kafka 配置端口

9094

用于代理间通信。但在 Kafka 连接中，您尝试通过端口

9092

连接到代理。

当您尝试连接到端口

9092

时，

localhost:9092

将作为元数据发送到 Kafka 连接以进行进一步通信，但

localhost:9092

在 docker 网络内不可用。使用下面的 Kafka 连接配置进行连接 -

kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9094
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

广告听众的概念有点棘手。如果您想了解有关听众和广告听众的更多信息，请参阅此链接。

Answer 2

我在尝试通过 docker compose 文件设置 Kafka 时遇到了同样的问题，经过几个小时的研究，我解决了这个问题：

在 docker 文件的 Kafka 服务上，我定义了一个名为 KAFKA_ADVERTISED_HOST_NAME 的配置字段，通过此配置，我们定义了客户端（生产者和消费者）应使用的地址/IP，以便与 Kafka 代理连接，所以我配置了它像这样：

      KAFKA_ADVERTISED_HOST_NAME: localhost

然后，当我再次启动 docker compose 文件时，客户端能够连接到代理，您可以看到 Kafka 日志，其中显示了下面屏幕截图中突出显示的消息：

之后，我在 Spring Boot 应用程序中进行配置，在 application.properties 文件中，我定义了该字段：

spring.kafka.bootstrap-servers=localhost:9092