如果第一个经纪人失败，卡夫卡消费者就无法消费

Question

我正在使用最新版本的kafka（kafka_2.12-1.0.0.tgz）。我已经设置了包含3个代理的简单集群（在每个实例的属性文件中只更改了broker.id = 1和listeners = PLAINTEXT：//：9092）。集群启动后，我使用以下命令创建了主题

./kafka-topics.sh --create    --zookeeper localhost:2181  --replication-factor 3     --partitions 13    --topic demo

然后使用以下命令启动kafka使用者和生产者

./kafka-console-producer.sh --topic  demo  --broker-list localhost:9094,localhost:9093,localhost:9092

./kafka-console-consumer.sh --group test --bootstrap-server localhost:9094,localhost:9093,localhost:9092  --topic demo

当所有经纪人都出现时，一切都很好。但是，如果我先杀死（通过启动顺序），经纪人消息将被发送给经纪人，但消费者无法接收任何消息。消息不会丢失。启动后，该代理消费者立即收到消息。

关闭代理实例后的消费者日志：

[2018-01-09 13：39：31,130] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：31,132] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点1的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：31,344] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：31,451] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点1的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：31,848] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：31,950] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点1的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：32,363] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：33,092] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：34,216] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点2147483646的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）[2018-01-09 13：39：34,218] WARN [Consumer clientId = consumer-1，groupId = test]偏移的异步自动提交{demo-0 = OffsetAndMetadata {offset = 3，metadata =''}，demo-1 = OffsetAndMetadata {offset = 3，metadata =''}，demo-2 = OffsetAndMetadata {offset = 2，metadata =''}，demo-3 = OffsetAndMetadata {offset = 2 ，metadata =''}，demo-4 = OffsetAndMetadata {offset = 1，metadata =''}，demo-5 = OffsetAndMetadata {offset = 1，metadata =''}，demo-6 = OffsetAndMetadata {offset = 3，metadata =''}，demo-7 = OffsetAndMetadata {offset = 2，metadata =''}，demo-8 = OffsetAndMetadata {offset = 3，metadata =''}，demo-9 = OffsetAndMetadata {offset = 2，metadata =' '}，demo-10 = OffsetAndMetadata {offset = 3，metadata =''}，demo-11 = OffsetAndMetadata {offset = 2，metadata =''}，demo-12 = OffsetAndMetadata {offset = 2，metadata =''}失败：偏移提交因可恢复异常而失败。您应该重试提交偏移量。潜在的错误是：协调员不可用。 （org.apache.kafka.clients.consumer.internals.ConsumerCoordinator）[2018-01-09 13：39：34,219] WARN [Consumer clientId = consumer-1，groupId = test]无法建立与节点1的连接。经纪人可能无法使用。 （org.apache.kafka.clients.NetworkClient）

再次启动遗失经纪人后记录消费者：

[2018-01-09 13：41：21,739]错误[消费者clientId = consumer-1，groupId = test]在偏移3处的分区demo-0上的偏移提交失败：这不是正确的协调器。 （org.apache.kafka.clients.consumer.internals.ConsumerCoordinator）[2018-01-09 13：41：21,739] WARN [Consumer clientId = consumer-1，groupId = test]偏移的异步自动提交{demo-0 = OffsetAndMetadata {offset = 3，metadata =''}，demo-1 = OffsetAndMetadata {offset = 3，metadata =''}，demo-2 = OffsetAndMetadata {offset = 2，metadata =''}，demo-3 = OffsetAndMetadata {offset = 2，metadata =''}，demo-4 = OffsetAndMetadata {offset = 1，metadata =''}，demo-5 = OffsetAndMetadata {offset = 1，metadata =''}，demo-6 = OffsetAndMetadata {offset = 3，metadata =''}，demo-7 = OffsetAndMetadata {offset = 2，metadata =''}，demo-8 = OffsetAndMetadata {offset = 3，metadata =''}，demo-9 = OffsetAndMetadata {offset = 2 ，metadata =''}，demo-10 = OffsetAndMetadata {offset = 3，metadata =''}，demo-11 = OffsetAndMetadata {offset = 2，metadata =''}，demo-12 = OffsetAndMetadata {offset = 2，metadata =''}}失败：偏移提交因可恢复异常而失败。您应该重试提交偏移量。潜在的错误是：这不是正确的协调员。 （org.apache.kafka.clients.consumer.internals.ConsumerCoordinator）[2018-01-09 13：41：22,353] ERROR [Consumer clientId = consumer-1，groupId = test]偏移提交在分区demo-0上失败3：这不是正确的协调员。 （org.apache.kafka.clients.consumer.internals.ConsumerCoordinator）[2018-01-09 13：41：22,354] WARN [Consumer clientId = consumer-1，groupId = test]偏移的异步自动提交{demo-0 = OffsetAndMetadata {offset = 3，metadata =''}，demo-1 = OffsetAndMetadata {offset = 3，metadata =''}，demo-2 = OffsetAndMetadata {offset = 2，metadata =''}，demo-3 = OffsetAndMetadata {offset = 2，metadata =''}，demo-4 = OffsetAndMetadata {offset = 1，metadata =''}，demo-5 = OffsetAndMetadata {offset = 1，metadata =''}，demo-6 = OffsetAndMetadata {offset = 3，metadata =''}，demo-7 = OffsetAndMetadata {offset = 2，metadata =''}，demo-8 = OffsetAndMetadata {offset = 3，metadata =''}，demo-9 = OffsetAndMetadata {offset = 2 ，metadata =''}，demo-10 = OffsetAndMetadata {offset = 3，metadata =''}，demo-11 = OffsetAndMetadata {offset = 3，metadata =''}，demo-12 = OffsetAndMetadata {offset = 2，metadata =''}}失败：偏移提交因可恢复异常而失败。您应该重试提交偏移量。潜在的错误是：这不是正确的协调员。 （org.apache.kafka.clients.consumer.internals.ConsumerCoordinator）

谢谢

Answer 1

尝试检查server - * .properties文件中的“offsets.topic.replication.factor”

例如：

############################# Internal Topic Settings       
# The replication factor for the group metadata internal topics    
# For anything other than development testing, a value greater than 1 is  recommended for to ensure availability such as 3.
offsets.topic.replication.factor=3

http://kafka.apache.org/documentation/#brokerconfigs

Answer 2

在KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR文件上使用yml解决了这个问题。

例如。在docker-swarm上使用2名工人。

environment:
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2