GKE 网关 API httproute 不适用于负载均衡器和应用程序之间的 https

问题描述 投票：0回答：2

我正在使用 GKE（版本 1.21.11-gke.1100）测试 Gateway API。我使用 

gatewayClassName: gke-l7-rilb
作为客户端和网关之间具有 TLS 的网关。 HTTPS 使用托管区域 SSL 证书在客户端和负载均衡器之间完美运行。

我有 2 个 Httproutes 引用 2 个 kube 服务 (

backendRefs
)。一项服务可通过 HTTP 访问，另一项可通过 HTTPS 访问（来自 Argo Workflows 项目的 argo 服务器服务，如果有帮助的话）。

当我使用 HTTP 创建引用服务的 httproute 时，GCP 负载均衡器后端服务将被创建并正常工作（正常）。

但是当我创建引用 argo-service 的 httproute 时，会创建一个 GCP 负载均衡器后端服务，但在端点协议设置为 HTTP 而不是 HTTPS 时无法工作（不健康）。您应该知道，我确保向 argo-server 服务添加注释 

cloud.google.com/app-protocols: '{"web":"HTTPS"}'
以在负载均衡器和 argo-server 应用程序之间启用 HTTPS。

如果我使用入口资源和相同的 argo 服务定义创建相同的 geatway api 配置，端点协议（GCP 负载均衡器后端服务）将正确设置为 HTTPS，并且完全健康且正常工作。

就像网关 API 的 httproute 一样，GKE 网关控制器没有考虑 

cloud.google.com/app-protocols
服务注释，尽管这里提到它与网关 API 相关。

编辑1：添加yaml文件

  • 网关：
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
    networking.gke.io/addresses: ""
    networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/firewalls: ""
    networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
    networking.gke.io/ssl-certificates: ""
    networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
  creationTimestamp: "2022-06-15T08:28:20Z"
  finalizers:
  - gateway.finalizer.networking.gke.io
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:addresses: {}
        f:gatewayClassName: {}
        f:listeners:
          .: {}
          k:{"name":"https"}:
            .: {}
            f:allowedRoutes:
              .: {}
              f:kinds: {}
              f:namespaces:
                .: {}
                f:from: {}
                f:selector:
                  .: {}
                  f:matchLabels:
                    .: {}
                    f:exposed: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:tls:
              .: {}
              f:mode: {}
              f:options:
                .: {}
                f:networking.gke.io/pre-shared-certs: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T08:28:20Z"
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:networking.gke.io/addresses: {}
          f:networking.gke.io/backend-services: {}
          f:networking.gke.io/firewalls: {}
          f:networking.gke.io/forwarding-rules: {}
          f:networking.gke.io/health-checks: {}
          f:networking.gke.io/last-reconcile-time: {}
          f:networking.gke.io/ssl-certificates: {}
          f:networking.gke.io/target-proxies: {}
          f:networking.gke.io/url-maps: {}
        f:finalizers:
          .: {}
          v:"gateway.finalizer.networking.gke.io": {}
      f:status:
        f:addresses: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T08:30:16Z"
  name: regional-internal-https
  namespace: exposition
  resourceVersion: "42337844"
  uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
  addresses:
  - type: NamedAddress
    value: dev-gateway-internal-lb-static-ip
  gatewayClassName: gke-l7-rilb
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            exposed: "true"
    name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
  addresses:
  - type: IPAddress
    value: 10.163.112.28
  conditions:
  - lastTransitionTime: "1970-01-01T00:00:00Z"
    message: Waiting for controller
    reason: NotReconciled
    status: Unknown
    type: Scheduled
  • http路由：
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
  creationTimestamp: "2022-06-15T12:27:04Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          .: {}
          f:hostnames: {}
          f:parentRefs: {}
          f:rules: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2022-06-15T12:27:04Z"
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:parents: {}
      manager: GoogleGKEGatewayController
      operation: Update
      time: "2022-06-15T12:29:02Z"
  name: argo-server
  namespace: argo
  resourceVersion: "42362026"
  uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
  hostnames:
    - argo-server.plat.dev.df.gcp.corp.modified.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: regional-internal-https
      namespace: exposition
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: argo-server
          port: 2746
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: RouteAccepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: ReconciliationSucceeded
          status: "True"
          type: Reconciled
      controllerName: networking.gke.io/gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: regional-internal-https
        namespace: exposition
  • 服务：
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"web":"HTTPS"}'
    cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
    cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{\"web\":\"HTTPS\"}","cloud.google.com/backend-config":"{\"default\": \"argo-server-backendconfig\"}","cloud.google.com/neg":"{\"ingress\": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
  creationTimestamp: "2022-06-15T11:44:07Z"
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cloud.google.com/app-protocols: {}
          f:cloud.google.com/backend-config: {}
          f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":2746,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T12:27:23Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T12:28:06Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg-status: {}
    manager: glbc
    operation: Update
    time: "2022-06-15T12:28:06Z"
  name: argo-server
  namespace: argo
  resourceVersion: "41692832"
  uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
  clusterIP: 10.163.247.121
  clusterIPs:
  - 10.163.247.121
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    port: 2746
    protocol: TCP
    targetPort: 2746
  selector:
    app: argo-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
kubernetes google-cloud-platform google-kubernetes-engine kubernetes-ingress kubernetes-gateway-api
2个回答
3
投票

我找到了一个解决方案，我认为这是一种解决方法。

  1. 使用

    networking.gke.io/app-protocols: '{"web":"HTTPS"}'
    注释而不是
    cloud.google.com/app-protocols: '{"web":"HTTPS"}'
    。此注释用于服务级别，其中 
    web
    是端口名称。这将在负载均衡器和应用程序之间启用 HTTPS （为指定 HTTPRoute 创建的后端服务的端点协议）。 这与
    gatewayClassName: gke-l7-rilb
    区域内部负载均衡器完美配合。

  2. 使用 

    cloud.google.com/v1 BackendConfig
    创建自定义运行状况检查，将类型设置为 HTTPS，将端口设置为 2746。更多详细信息请参见此处 https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features #direct_health 通过入口，GCE 入口控制器会自动从应用程序就绪探针创建此健康检查，但显然此功能尚未在 GKE 网关控制器中实现。

  3. 确保您的防火墙规则允许在 2746 端口上进行 Google Cloud 运行状况检查的入口流量通过入口，GCE 入口控制器会自动创建所需的防火墙规则，但显然此功能尚未在 GKE 网关控制器中实现。

最后我说这是一个解决方法，因为我想象并希望 GKE 网关控制器的未来版本能够解决我上面提到的 3 个问题或要点。

0
投票

对我来说同样的问题：如何通过使用 GKE 集群中的 Kuberenetes 资源HTTPRoute（使用 gatewayClass：gke-l7-gxlb）来允许负载均衡器和应用程序/k8s-pod 之间进行HTTPS通信：

解决方案：需要在提到的 kubernetes service: 中设置字段“appProtocol: HTTPS

” 
apiVersion: v1
kind: Service
metadata:
name: your-service
labels:
    app.kubernetes.io/name: your-service
spec:
type: ClusterIP
ports:
   - name: port-https
     port: 443
     targetPort: 8080
     protocol: TCP
     appProtocol: HTTPS
 selector:
    app.kubernetes.io/name: your-app

之后，您可以在后端服务的 GCP 控制台/负载均衡器详细信息中看到“端点协议”为“HTTPS”：

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.