filebeat to kafka : 无法连接到代理。

问题描述 投票:0回答:1

我是在apache环境下的新手,目前正在尝试从filebeat producer发送日志数据到kafka broker。

环境。

  1. kafka 2.11 (通过ambari安装)
  2. filebeat 7.4.2 (windows)

我试着从filebeat发送日志到ambari,我已经启动了kafka服务器,并创建了名为 "test "的主题,它被列在-list上。我对kafka代理的端口很困惑。在一些教程中,我看到他们使用的是9092而不是2181。那么现在,我应该用什么端口从filebeat发送日志?

这里是我的filebeat.conf

filebeat.inputs:
    - type: log
      paths:
       - C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
  hosts: ["x.x.x.x:9092"]
  topic: "test"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

结果

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)

2020-06-10T09:00:32.466+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)

2020-06-10T09:00:34.729+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)

2020-06-10T09:00:36.989+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

这让我怀疑我是否真的得到了9092端口。所以我检查了server.properties.Some,我最关心的是。

port=6667
listeners=PLAINTEXT://x.x.x.x:6667

所以我再次尝试做filebeat. conf,并改变端口9092到6667,这里是结果

2020-06-10T09:18:01.448+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667

2020-06-10T09:18:01.450+0700    INFO    kafka/log.go:53 producer/broker/1001 starting up

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [open] on test/0

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/leader/test/0 selected broker 1001

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.452+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:264     finished kafka batch
2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:278     Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [normal]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 abandoning broker 1001

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/broker/1001 shut down

疑问

怎么了?应该使用哪个端口?各个端口的用途是什么?

任何回应都会非常感激。谢谢你的回答

更新

据此 源头 正确的源码是6667,因为kafka是通过ambari安装的。

apache apache-kafka apache-zookeeper hadoop2 filebeat
1个回答
1
投票

在第一种情况下,就像你说的,broker可能是在6667上启动的,因此在9092上没有进程运行。

2020-06-10T09:18:01.451+0700 INFO kafkalog.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host.

其次,当你提到 advertised.listeners 财产,你应该确保你提到的IP在你的网站上。advertised.listeners 是分配给该计算机的IP。您不能指定 1.1.1.1:9092 (仅举一些例子)。

执行 ifconfig (linux)。ipconfig (windows),并看到你的机器在网络接口上的IP,可以从你的应用程序机器上访问。大多乙太

这个IP必须可以从你运行应用的机器上访问,所以你的应用所运行的机器应该可以解析这个IP。你可能还想检查你的Kafka代理和你正在运行应用程序的机器之间的网络连接。

© www.soinside.com 2019 - 2024. All rights reserved.