Storm kafka警告消息有多个任务比分区一些任务将空闲

Question

我有风暴拓扑，使用KafkaSpout接收来自Kafka经纪人的消息。但我没有看到喷口处理任何消息。在工作日志中，我找到了以下代码段。风暴版-0.9.3

2017-12-13T18:43:49.598+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing
2017-12-13T18:43:50.111+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:50.113+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing

我不确定为什么我会收到这样的警告信息。我尝试使用kafka-topic cli验证kafka主题，但我找到了3个分区。

./kafka-topics.sh  --zookeeper xxx.xxx.xxx.xxx:2181 --topic topic-name  --describe
Topic:topic-name PartitionCount:3  ReplicationFactor:1 Configs:
  Topic: topic-name  Partition: 0  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 1  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 2  Leader: 2 Replicas: 2 Isr: 2

我在这里做错了什么或者可能是没有收到消息的可能问题？

Answer 1

注意这些线条

2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle

它无法从Zookeeper中读取分区列表。当您使用storm-kafka KafkaSpout时，您需要将其配置为连接到Kafka集群的Zookeeper。你应该通过https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/ZkHosts.java#L33在SpoutConfig中设置它。 zookeeper连接字符串语法与您在终端中传递给kafka-topics.sh的语法相同。