无法联系 nginx 部署和 LoadBalancer 服务背后的服务

Question

这样做的最终目标是将

nginx

放在我的

redis-sentinel

pod 前面，这样我就可以使用我们的公司 SSL 证书来保护它们。

Redis 本身已启动并经过测试。我遇到的问题是

nginx

部署和相关服务。

最新更新：@Vahid 建议我使用 ingress 代替。我正在使用

MicroK8s

并且入口已启用。

由于我正在使用

Microk8s

我认为我需要更新

nginx-load-balancer-microk8s-conf

文件，所以这就是我所做的：

kubectl get configmap --namespace ingress

获取配置映射然后

kubectl edit configmap nginx-load-balancer-microk8s-conf --namespace ingress

编辑它 - 我添加了：

data:
  tcp-services: |-
    6379: "default/redis-service:6379:redis-ssl"
    26379: "default/redis-sentinel-service:26379:redis-ssl"

并且重新滚动了

daemonset

。

接下来，我将相同的

secrets

添加到

ingress

命名空间。

然后是

service.yaml

：

apiVersion: v1
kind: Service
metadata:
  name: redis-service-lb
  namespace: redis
spec:
  type: LoadBalancer
  loadBalancerIP: 10.250.0.44
  ports:
  - port: 6379
    targetPort: 6379
    protocol: TCP
    name: tcp-redis
  - port: 26379
    targetPort: 26379
    protocol: TCP
    name: tcp-redis-sentinel
  selector:
    app: redis

看着

kubectl get events

我可以看到一切都很顺利：

44s         Normal    IPAllocated        service/redis-sentinel-service-lb   Assigned IP ["10.250.0.41"]

服务正在运行：

redis-service-lb          LoadBalancer   10.152.183.56    10.250.0.44   6379:31301/TCP,26379:32502/TCP   5m49s

也就是说 - 仍然无法使用

nc

- 或

redis-cli

建立连接...

我确定我搞错了：这是我第一次与

ingress

跳舞...

历史帖子如下

我们将从安装为

nginx.conf

的

configmap

文件开始：

server {
    listen 6379 ssl;
    server_name ki44.MyDomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    sendfile        on;
    keepalive_timeout  65;

    ssl_protocols                               TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers                   on;
    ssl_certificate                             /etc/nginx/certs/tls.crt;
    ssl_certificate_key                         /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://redis-sentinel.redis.svc.cluster.local:6379;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 26379 ssl;
    server_name ki44.MyDomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    ssl_protocols                               TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers                   on;
    ssl_certificate                             /etc/nginx/certs/tls.crt;
    ssl_certificate_key                         /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://redis-sentinel.redis.svc.cluster.local:23679;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

必要的证书也安装为

secrets

。 Pod 启动时零错误。

接下来是我的

service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-nginx
  namespace: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-proxy
  template:
    metadata:
      labels:
        app: nginx-proxy
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 6379
          name: redis
          hostPort: 6379
        - containerPort: 26379
          name: sentinel
          hostPort: 26379
        volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d
        - name: cert-volume
          mountPath: /etc/nginx/certs
      volumes:
      - name: config-volume
        configMap:
          name: nginx-config
      - name: cert-volume
        secret:
          secretName: redis-ssl
      nodeSelector:
        location: internal
        type: worker
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis
spec:
  type: LoadBalancer
  loadBalancerIP: 10.250.0.44
  ports:
    - port: 6379
      name: redis
      targetPort: 6379
      protocol: TCP
    - port: 26379
      name: sentinel
      targetPort: 26379
      protocol: TCP
  selector:
    app: redis-nginx

我部署了这个，

nginx

pod 启动得很好，并且我通过炮击它（并在其中安装

nc

）确认我能够到达指定的适当端口。

一切都看起来很高兴，但我无法使用

nc

联系任一端口：

nc -zv ki44.MyDomain.com 6379

和

nc -zv ki44.MyDomain.com 26379

让我无处可去。

这是

kubectl get services

的输出

NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
redis-sentinel-headless   ClusterIP      None             <none>        6379/TCP,26379/TCP               3h30m
redis-sentinel            ClusterIP      10.152.183.130   <none>        6379/TCP,26379/TCP               3h30m
redis                     LoadBalancer   10.152.183.218   10.250.0.44   6379:30420/TCP,26379:30154/TCP   55m

NB：

KI44.MyDomain.com

确实确实决心

10.250.0.44

。

跑步

kubectl describe pod <podName>

给我：

Name:             redis-nginx-7595bdd458-7grll
Namespace:        redis
Priority:         0
Service Account:  default
Node:             kwi01.md.local/10.250.0.147
Start Time:       Sun, 12 May 2024 12:40:16 -0400
Labels:           app=nginx-proxy
                  pod-template-hash=7595bdd458
Annotations:      cni.projectcalico.org/containerID: cd6b7cf612081b0fb62e98f483d97a3f2d25d40b12e0321b70786158d7e8b7a9
                  cni.projectcalico.org/podIP: 10.1.40.48/32
                  cni.projectcalico.org/podIPs: 10.1.40.48/32
Status:           Running
IP:               10.1.40.48
IPs:
  IP:           10.1.40.48
Controlled By:  ReplicaSet/redis-nginx-7595bdd458
Containers:
  nginx:
    Container ID:   containerd://b224262eadc355bfc565a8654f65760f2b2cd5444baab353f15425666e718cad
    Image:          nginx:latest
    Image ID:       docker.io/library/nginx@sha256:32e76d4f34f80e479964a0fbd4c5b4f6967b5322c8d004e9cf0cb81c93510766
    Ports:          6379/TCP, 26379/TCP
    Host Ports:     6379/TCP, 26379/TCP
    State:          Running
      Started:      Sun, 12 May 2024 12:40:17 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/nginx/certs from cert-volume (rw)
      /etc/nginx/conf.d from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d865w (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nginx-config
    Optional:  false
  cert-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  redis-ssl
    Optional:    false
  kube-api-access-d865w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              location=internal
                             type=worker
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

来自 Pod 的日志：

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/05/12 16:40:17 [notice] 1#1: using the "epoll" event method
2024/05/12 16:40:17 [notice] 1#1: nginx/1.25.5
2024/05/12 16:40:17 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/05/12 16:40:17 [notice] 1#1: OS: Linux 5.15.0-102-generic
2024/05/12 16:40:17 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:65536
2024/05/12 16:40:17 [notice] 1#1: start worker processes
2024/05/12 16:40:17 [notice] 1#1: start worker process 21
2024/05/12 16:40:17 [notice] 1#1: start worker process 22
2024/05/12 16:40:17 [notice] 1#1: start worker process 23
2024/05/12 16:40:17 [notice] 1#1: start worker process 24
2024/05/12 16:40:17 [notice] 1#1: start worker process 25
2024/05/12 16:40:17 [notice] 1#1: start worker process 26
2024/05/12 16:40:17 [notice] 1#1: start worker process 27
2024/05/12 16:40:17 [notice] 1#1: start worker process 28

运行

kubectl get events

也没有显示任何奇怪的东西：

34s         Normal    Killing             pod/redis-nginx-7595bdd458-7grll    Stopping container nginx
20s         Normal    IPAllocated         service/redis                       Assigned IP ["10.250.0.44"]
20s         Normal    ScalingReplicaSet   deployment/redis-nginx              Scaled up replica set redis-nginx-7595bdd458 to 1
20s         Normal    SuccessfulCreate    replicaset/redis-nginx-7595bdd458   Created pod: redis-nginx-7595bdd458-lwcsg
19s         Normal    Scheduled           pod/redis-nginx-7595bdd458-lwcsg    Successfully assigned redis/redis-nginx-7595bdd458-lwcsg to kwi01.md.local
18s         Normal    Pulling             pod/redis-nginx-7595bdd458-lwcsg    Pulling image "nginx:latest"
18s         Normal    Pulled              pod/redis-nginx-7595bdd458-lwcsg    Successfully pulled image "nginx:latest" in 244.860879ms (244.883758ms including waiting)
18s         Normal    Created             pod/redis-nginx-7595bdd458-lwcsg    Created container nginx
18s         Normal    Started             pod/redis-nginx-7595bdd458-lwcsg    Started container nginx

我尝试过的事情

将 proxy_pass

http://redis-sentinel.redis.svc.cluster.local.*

更改为

proxy_pass http://redis-sentinel-headless.redis.svc.cluster.local:*

。

在我的
```
hostPort
```
中添加和删除
```
Deployment
```
作业。
删除
```
SSL
```
设置并尝试不使用加密。
仅使用一个端口进行测试 -
```
6379
```
。
在同一命名空间中创建一个新的 pod，安装
```
curl
```
，然后使用它通过内部 IP 地址（例如 10.1.40.48）连接到
```
nginx
```
代理。然后我在
```
nginx
```
pod 本身的日志中看到了此尝试，所以我知道
```
nginx
```
代理确实正在运行。

我所知道的事情

这不是防火墙问题。
这不是网络连接问题。
```
ki44.MyDomain.com
```
上没有其他内容正在收听。从
```
nginx
```
吊舱内，我可以看到
```
637
```
和
```
26379
```
都在监听。我还能够成功执行
```
nc -zv redis-sentinel.redis.svc.cluster.local:6379
```
（以及所有其他衍生品）。
我还有其他几个
```
LoadBalancer
```
服务在集群上运行得很好 - 尽管这是我在其前面使用
```
nginx
```
完成的第一个服务。
我很困惑。😕

Answer 1

将您的服务

spec.type

从

LoadBalancer

更改为

ClusterIP

。现在，您的

Redis

仅在集群内公开。接下来，创建您的 Ingress 资源，例如

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-redis
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: "/redis"
        backend:
          service:
            name: redis
            port:
              number: 6379
      - pathType: Prefix
        path: "/sentinel"
        backend:
          service:
            name: redis
            port:
              number: 26379

这会将您的服务暴露给外部。

注意主机。如果您想访问您的入口，您的请求应该在

host

标头中包含此内容。假设您的集群位于您的计算机上（本地）。您可以通过

进行测试

curl -H "Host: foo.bar.com" http://127.0.0.1/redis

无法联系 nginx 部署和 LoadBalancer 服务背后的服务

问题描述投票：0回答：1

历史帖子如下

我尝试过的事情

我所知道的事情

1个回答

最新问题

无法联系 nginx 部署和 LoadBalancer 服务背后的服务

问题描述 投票：0回答：1

历史帖子如下

我尝试过的事情

我所知道的事情

1个回答

最新问题

问题描述投票：0回答：1