通过terraform部署Helm Chart时无法传递服务注解

Question

我看到了一些关于如何在通过 terraform 部署 Helm Chart 时传递注释的示例，但没有一个按预期工作，在这种情况下，我尝试创建一个在特定子网上分配私有 IP 的服务，但相反，它的创建公共IP。 我的地形文件：

locals {
  helm_general = {
    # Reference values
    # https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
    elasticsearch = {
      name      = "elasticsearch"
      chart     = "elastic/elasticsearch"
      tag       = "7.14.0"
      namespace = "elasticsearch"
      set = [
        {
          name  = "nodeSelector.agentpool"
          value = "general"
        },      
        {
          name  = "replicas"
          value = "1"
        },
        {
          name  = "minimumMasterNodes"
          value = "1"
        },
        {
          name  = "image"
          value = "docker.elastic.co/elasticsearch/elasticsearch"
        },
        {
          name  = "imageTag"
          value = "7.14.0"
        },
        {
          name  = "resources.requests.cpu"
          value = "10m"
        },
        {
          name  = "resources.requests.memory"
          value = "128Mi"
        },
        {
          name  = "volumeClaimTemplate.reosources.requests.storage"
          value = "4Gi"
        },
        {
          name  = "persistence.enabled"
          value = "false"
        },
        {
          name  = "service.type"
          value = "LoadBalancer"
        },
        {
          name  = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
          value = "true"
        },
        {
          name  = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal-subnet"
          value = "somesubnet"
        },          
      ]
      timeout = "900"
    }
  }
}

头盔部署

resource "helm_release" "helm" {
  provider  = helm.general
  for_each  = local.helm_general
  name      = each.value.name
  chart     = each.value.chart
  namespace = format(each.value.namespace)
  dynamic "set" {
    iterator = item
    for_each = each.value.set == null ? [] : each.value.set

    content {
      name  = item.value.name
      value = item.value.value
    }
  }
  depends_on = [kubernetes_namespace.general]
}

计划/应用输出

并且当前部署的是公网ip而不是私有ip：

Namespace:                elasticsearch                                                     
Labels:                   app=elasticsearch-master                                          
                          app.kubernetes.io/managed-by=Helm                                 
                          chart=elasticsearch                                               
                          heritage=Helm                                                     
                          release=elasticsearch                                             
Annotations:              meta.helm.sh/release-name: elasticsearch                          
                          meta.helm.sh/release-namespace: elasticsearch                     
Selector:                 app=elasticsearch-master,chart=elasticsearch,release=elasticsearch
Type:                     LoadBalancer                                                      
IP Families:              <none>                                                            
IP:                       xx                                                      
IPs:                      xxx                                                     
LoadBalancer Ingress:     redacted public ip                                                      
Port:                     http  9200/TCP                                                    
TargetPort:               9200/TCP                                                          
NodePort:                 http  32083/TCP                                                   
Endpoints:                                                                                  
Port:                     transport  9300/TCP                                               
TargetPort:               9300/TCP                                                          
NodePort:                 transport  32638/TCP                                              
Endpoints:                                                                                  
Session Affinity:         None                                                              
External Traffic Policy:  Cluster                                                           
Events:                                                                                     
  Type    Reason                Age        From                Message                      
  ----    ------                ----       ----                -------                      
  Normal  EnsuringLoadBalancer  1s         service-controller  Ensuring load balancer       
  Normal  EnsuredLoadBalancer   <invalid>  service-controller  Ensured load balancer        

我一直关注的参考文献：

https://github.com/hashicorp/terraform-provider-helm/issues/125 https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release

编辑：

错误信息：

Error: unable to decode "": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|nternal":true},"labe|..., bigger context
..|beta.kubernetes.io/azure-load-balancer-internal":true},"labels":{"app":"elasticsearch-master","chart|...

  with helm_release.helm["elasticsearch"],
  on aks-general-helm.tf line 1, in resource "helm_release" "helm":
   1: resource "helm_release" "helm" {

Answer 1

我刚刚遇到了类似的问题，这对我有用：

{
 name  = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
 value = "true"
},

我认为问题在于它是如何连接的。 Elastic 官方的图表清单中的服务是

service.annotations:{}

，因此您需要附加

.service

，然后使用

\\

。

Answer 2

无需转义字符并保持原始 YAML 格式的一种方法是使用 helm_release 资源的

values

属性。我很想知道是否有一个特定的用例不首先使用它。

resource "helm_release" "helm" {
  provider  = helm.general
  for_each  = local.helm_general
  name      = each.value.name
  chart     = each.value.chart
  namespace = format(each.value.namespace)

  values = each.value.values ##CHANGE IS HERE ##

  dynamic "set" {
    iterator = item
    for_each = each.value.set == null ? [] : each.value.set

    content {
      name  = item.value.name
      value = item.value.value
    }
  }
  depends_on = [kubernetes_namespace.general]
}

您的情况下的本地将调整为以下，您仍然可以保留对任何地形资源依赖或任何其他逻辑原因的设置。

locals {
  helm_general = {
    # Reference values
    # https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
    elasticsearch = {
      [...]
      values = [file("${path.module}/elasticsearch-values.yaml")]
      [...]
    }
  }
}

必须在存在此 terraform 配置的同一路径（可以根据

elasticsearch-values.yaml

使用任何相对路径进行调整）位置有一个新文件

local.helm_general.elasticsearch.values

。

# Reference values can be adapted as per the upstream chart.
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "somesubnet"

Answer 3

这是我向 ArgoCD k8s 服务添加注释的方法：

resource "helm_release" "argocd" {
...
  dynamic "set" {
    for_each = local.argocd_service_annotations
    content {
      // server.service.annotations
      name = "argo-cd.server.service.annotations.${replace(set.key, ".", "\\.")}"
      value = set.value
    }
  }
}

  argocd_service_annotations = {
    "external-dns.alpha.kubernetes.io/hostname" : "myhost.com"
  }

结果：

argo-cd:
  server:
    service:
      annotations:
        external-dns.alpha.kubernetes.io/hostname: myhost.com

Answer 4

如果你有多个，你可以尝试这样的方法，将注释作为地图传递并使用设置它们

  dynamic "set" {
    for_each = var.ingress_annotations
    content {
      name = replace(set.key, ".", "\\.")
      value = set.value
    }
  }