Kafka Connect AWS S3接收器连接器未从主题中读取

问题描述 投票:1回答:1

我有一个简单的独立S3接收器连接器。这是工作程序配置属性的相关部分:

plugin.path = <plugins directory>
bootstrap.servers = <List of servers on Amazon MKS>
security.protocol = SSL
...

当我将其连接到本地运行的Kafka时,效果很好。但是,当我将其连接到AWS上的Kafka代理(使用SSL)时,它不会消耗任何东西。没有错误,没事。仿佛主题为空:

[2020-01-30 10:50:03,597] INFO Started S3 connector task with assigned partitions: [] (io.confluent.connect.s3.S3SinkTask:116)
[2020-01-30 10:50:03,598] INFO WorkerSinkTask{id=xxx} Sink task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:302)

当我在connect-log4j.properties中启用DEBUG模式时,我开始看到很多错误消息:

Completed connection to node -2. Fetching API versions. (org.apache.kafka.clients.NetworkClient:914)
Initiating API versions fetch from node -2. (org.apache.kafka.clients.NetworkClient:928)
Connection with YYY disconnected (org.apache.kafka.common.network.Selector:607)
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
...
Node -2 disconnected. (org.apache.kafka.clients.NetworkClient:894)
Initialize connection to node XXX (id: -3 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient:1125)
Initiating connection to node XXX (id: -3 rack: null) using address XXX (org.apache.kafka.clients.NetworkClient:956)

我是否缺少使用SSL配置的功能?请注意,仅设置“ security.protocol = SSL”,手动创建的org.apache.kafka.clients.consumer.KafkaConsumer即可成功读取此主题。

编辑:这是连接器属性:

name = my-connector
connector.class = io.confluent.connect.s3.S3SinkConnector
topics = some_topic
timestamp.extractor = Record
locale = de_DE
timezone = UTC
storage.class = io.confluent.connect.s3.storage.S3Storage
partitioner.class = io.confluent.connect.storage.partitioner.HourlyPartitioner
format.class = io.confluent.connect.s3.format.bytearray.ByteArrayFormat
s3.bucket.name = some-s3-bucket
s3.compression.type = gzip
flush.size = 3
s3.region = eu-central-1
apache-kafka apache-kafka-connect confluent
1个回答
1
投票

我有一个类似的问题,在我另外为消费者指定了安全协议之后(除了全局协议之外,解决了这个问题:所以只需添加

© www.soinside.com 2019 - 2024. All rights reserved.