Kafka 连接 Debezium Mongo 源连接器无法连接到 MongoDb 实例

Question

以下命令用于在 Docker 中设置 Zookeeper、Kafka、Schema-registry、Kafka-connect 和 MongoDb。 Kakfa-connect 集群似乎没有连接到 MongoDb，它一直显示

Connection refused

。所有容器都部署在名为 confluent 的 docker 网络中。

# create confluent network \ 
docker network create confluent && \

# create Zookeeper \ 
docker run -d \
    --name=zookeeper \
    --net=confluent \
    -p 2181:2181 \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:5.4.0 && \ 

# create kafka \ 
docker run -d \
    --name=kafka \
    --net=confluent \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:5.4.0 && \

# create schema-registry \ 
docker run -d \
  --name=schema-registry \
  --net=confluent \
  -p 8081:8081 \
  -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181 \
  -e SCHEMA_REGISTRY_HOST_NAME=schema-registry \
  -e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 \
  confluentinc/cp-schema-registry:5.4.0 && \

# create kafka-connect \ 
docker run -d \
  --name=kafka-connect \
  --net=confluent \
  -p 8083:8083 \
  -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
  -e CONNECT_REST_PORT=8083 \
  -e CONNECT_GROUP_ID="quickstart" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-config" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="quickstart-status" \
  -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_REST_ADVERTISED_HOST_NAME="kafka-connect" \
  -e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
  -e CONNECT_PLUGIN_PATH=/usr/share/java \
  -e CONNECT_REST_HOST_NAME="kafka-connect" \
  -v /tmp/quickstart/file:/tmp/quickstart \
  confluentinc/cp-kafka-connect:5.4.0 && \

# create mongo \ 
docker pull mongo && \
docker run -d --name=mongodb --net=confluent  -v $(pwd)/data/db:/data/db -p 27017:27017 -d mongo  --replSet debezium && \
docker exec -it mongodb mongosh --eval 'rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]})'

我正在使用以下命令设置 Kafka Debezium MongoDb 连接器

docker exec -it kafka-connect bash
confluent-hub install debezium/debezium-connector-mongodb:0.9.4
docker restart kafka-connect

最后，我在 mongosh 中使用以下内容为 Debezium 创建用户

use admin
db.createUser(
 {
  user: "debezium",
  pwd: "dbz",
  roles: ["dbOwner"]
 })

现在我正在尝试使用以下 json 文件设置 MongoDb Debezium Kafka 连接器

{
    "name": "inventory-connector",
    "config": {
        "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
        "tasks.max": "1",
        "mongodb.hosts": "debezium/127.0.0.1:27017",
        "mongodb.name": "dbserver1",
        "mongodb.user": "debezium",
        "mongodb.password": "dbz"
    }
}

curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mongodb.json

现在，当我触发上述 curl 命令时，这是我在 Kafka-connect docker 容器中看到的错误消息

[2023-04-29 12:58:26,106] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
    connector.class = io.debezium.connector.mongodb.MongoDbConnector
    connector.class = io.debezium.connector.mongodb.MongoDbConnector
[2023-04-29 12:58:26,132] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2023-04-29 12:58:26,144] INFO Cluster description not yet available. Waiting for 30000 ms before timing out (org.mongodb.driver.cluster)

[2023-04-29 12:58:43,303] DEBUG Closing connection connectionId{localValue:36} (org.mongodb.driver.connection)
[2023-04-29 12:58:43,304] DEBUG Updating cluster description to  
{type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception=
{com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.ConnectException: Connection refused (Connection refused)}}] 
(org.mongodb.driver.cluster)

Kafka-connect 集群好像无法连接到MongoDb。知道发生了什么事吗？

Answer 1

debezium/127.0.0.1:27017

带有斜杠的主机名不是有效的，并且您的容器都没有命名为 debezium。你想连接到 Mongo 容器。

你设置

--name=mongodv

。使用它而不是尝试让 Connect 容器启动一个地址指向自身的 Mongo 客户端，正如错误所说的那样。