aws msk 上的 FilePulse 源连接器 - TimeoutException

问题描述 投票:0回答:1

我目前正在尝试使用 FilePulse Source Connector,但不断收到这些错误

org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata,.... 
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
apache-kafka apache-kafka-connect
1个回答
0
投票

我有一个非常相似的问题,这是我的配置和堆栈跟踪

配置:

aws.s3.bucket.name = my-bucket
aws.s3.bucket.prefix = kafka-test-data
aws.s3.region = eu-west-1
connector.class = io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
file.filter.regex.pattern = .*\\.csv$
filters = ParseCSVLine
filters.ParseCSVLine.auto.generate.column.names = false
filters.ParseCSVLine.extract.column.name = headers
filters.ParseCSVLine.separator = ,
filters.ParseCSVLine.trim.column = true
filters.ParseCSVLine.type = io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter
fs.cleanup.policy.class = io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy
fs.listing.class = io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing
fs.listing.filters = io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter
fs.listing.interval.ms = 10000
offset.attributes.string = uri
offset.strategy = name
skip.headers = 1
tasks.file.status.storage.bootstrap.servers = blah-blah.kafka.eu-west-1.amazonaws.com:9092
tasks.file.status.storage.topic = connect-file-pulse-status
tasks.file.status.storage.topic.partitions = 3
tasks.file.status.storage.topic.replication.factor = 1
tasks.max = 1
tasks.reader.class = io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3RowFileInputReader
topic = my-test-topic-source

堆栈跟踪:

[Worker-069a1a0b439747c66] [2023-09-02 06:17:19,367] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
[Worker-069a1a0b439747c66] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1693635439366, tries=1, nextAllowedTryMs=1693635439467) timed out at 1693635439367 after 1 attempt(s)
[Worker-069a1a0b439747c66] Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,361] INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,361] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
[Worker-069a1a0b439747c66] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1693635469367, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
[Worker-069a1a0b439747c66] Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,366] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:668)
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,366] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:672)
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,366] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:678)
[Worker-069a1a0b439747c66] [2023-09-02 06:17:49,366] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
[Worker-069a1a0b439747c66] org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
[Worker-069a1a0b439747c66]  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
[Worker-069a1a0b439747c66]  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
[Worker-069a1a0b439747c66]  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
[Worker-069a1a0b439747c66]  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
[Worker-069a1a0b439747c66] Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1693635469359, tries=1, nextAllowedTryMs=1693635469460) timed out at 1693635469360 after 1 attempt(s)
[Worker-069a1a0b439747c66]  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
[Worker-069a1a0b439747c66]  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
[Worker-069a1a0b439747c66]  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
[Worker-069a1a0b439747c66]  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
[Worker-069a1a0b439747c66]  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
[Worker-069a1a0b439747c66]  ... 3 more
[Worker-069a1a0b439747c66] Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1693635469359, tries=1, nextAllowedTryMs=1693635469460) timed out at 1693635469360 after 1 attempt(s)
[Worker-069a1a0b439747c66] Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
[Worker-069a1a0b439747c66] MSK Connect encountered errors and failed.
© www.soinside.com 2019 - 2024. All rights reserved.