如何在Kubernetes外部连接到Kafka

Question

我使用Yolean/kubernetes-kafka和Helm chart成功地将Kafka部署到当地Docker（gcp＆minikube）上的Kubernetes

使用此python脚本从集群内成功测试主题生成：

#!/usr/bin/env python

from kafka import KafkaConsumer, KafkaProducer

KAFKA_TOPIC = 'demo'
# KAFKA_BROKERS = 'localhost:32400' # see step 1

# from inside the cluster in a different namespace
# KAFKA_BROKERS = 'bootstrap.kafka.svc.cluster.local:9092'

KAFKA_BROKERS = 'kafka.kafka.svc.cluster.local:9092'

print('KAFKA_BROKERS: ' + KAFKA_BROKERS)

producer = KafkaProducer(bootstrap_servers=KAFKA_BROKERS)


messages = [b'hello kafka', b'Falanga', b'3 test messages']


for m in messages:
    print(f"sending: {m}")
    producer.send(KAFKA_TOPIC, m)

producer.flush()

掌舵我使用此选项启用外部使用：

helm install --name kafka --set external.enabled=true --namespace kafka incubator/kafka

在我使用的原始回购中：

kubectl apply -f ./outside-0.yml

生成的服务具有端点和节点端口，但脚本无法在群集外部运行。

这是原始服务（分支主）

➜  ~ kubectl describe svc outside-0 --namespace kafka
Name:                     outside-0
Namespace:                kafka
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-        configuration={"apiVersion":"v1","kind":"Service","metadata":    {"annotations":{},"name":"outside-0","namespace":"kafka"},"spec":{"ports":    [{"nodePort":32400,"port":3240...
Selector:                 app=kafka,kafka-broker-id=0
Type:                     NodePort
IP:                       10.99.171.133
LoadBalancer Ingress:     localhost
Port:                     <unset>  32400/TCP
TargetPort:               9094/TCP
NodePort:                 <unset>  32400/TCP
Endpoints:                10.1.3.63:9094
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

这是掌舵服务描述：

Name:                     kafka-0-external
Namespace:                kafka
Labels:                   app=kafka
                          chart=kafka-0.9.2
                          heritage=Tiller
                          pod=kafka-0
                          release=kafka
Annotations:                  dns.alpha.kubernetes.io/internal=kafka.cluster.local
                      external-    dns.alpha.kubernetes.io/hostname=kafka.cluster.local
Selector:                 app=kafka,pod=kafka-0,release=kafka
Type:                     NodePort
IP:                       10.103.70.223
LoadBalancer Ingress:     localhost
Port:                     external-broker  19092/TCP
TargetPort:               31090/TCP
NodePort:                 external-broker  31090/TCP
Endpoints:                10.1.2.231:31090
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

本地docker节点没有externalIP字段：

kubectl describe node docker-for-desktop | grep IP
InternalIP:  192.168.65.3

我按照外面的Readme指示，即

1. 将hotsPot添加到50 kafka statefulset 9094端口
2. 在10broker-config中添加节点端口发现

并发现本地docker节点没有externalIP字段

如何从docker上的集群外部连接到kafka？这是否适用于GKE或其他部署？

Answer 1

该服务将pod暴露给内部Kubernetes网络。为了将服务（暴露pod）暴露给互联网，您需要设置指向服务的Ingress。

Ingresses基本上相当于Kubernetes的Apache / Nginx。您可以通过以下URL了解如何执行此操作：

https://kubernetes.io/docs/concepts/services-networking/ingress/

或者，您可以通过将service type定义为NodePort并将特定端口分配给它来在节点网络上公开pod。它应该类似于以下内容：

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 31090
      name: http