Prometheus Alertmanager 向多个 Slack 通道发送通知

问题描述 投票：0回答：5

我们有两个不同的团队致力于不同的应用程序。我想通过使用相同的警报表达式将警报通知发送到不同的 Slack 通道。我找到了一些例子，但不明白在尝试添加新路线时使用

receiver: 'default'
的主要原因是什么？它的作用是什么？如果我更改它会产生什么影响？

同时，如果您能帮助我如何将通知发送到多个松弛通道，我将不胜感激。我尝试过新的通道。

当前alertmanager.yml

receivers:
  - name: 'team-1'
    slack_configs:
    - api_url: 'https://hooks.slack.com/services/1'
      channel: '#hub-alerts'
route:
  group_wait: 10s
  group_interval: 5m
  receiver: 'team-1'
  repeat_interval: 1h
  group_by: [datacenter]

新的alertmanager.yml

alertmanager.yml:
    receivers:
      - name: 'team-1'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/1'
          channel: '#channel-1'
          send_resolved: true
      
      - name: 'team-2'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/2'
          channel: '#channel-2'
          send_resolved: true

route:
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h
  group_by: [datacenter]
  receiver: 'default'
  routes:
  - receiver: 'team-1'
  - receiver: 'team-2'
monitoring prometheus prometheus-alertmanager
5个回答
23
投票

您需要将路线上的 continue 属性设置为 true。默认情况下为 false。

AlertManager 的默认行为是遍历您的路由以查找匹配项，并在找到匹配项的第一个节点处退出。

您想要做的是在比赛中发出警报，然后继续搜索其他比赛并激发这些比赛。

相关文档部分：https://prometheus.io/docs/alerting/latest/configuration/#route

使用此的示例： https://awesome-prometheus-alerts.grep.to/alertmanager.html

内嵌上面的示例，以防它损坏。

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "slack"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "slack"
      group_wait: 10s
      match_re:
        severity: critical|warning
      continue: true

    - receiver: "pager"
      group_wait: 10s
      match_re:
        severity: critical
      continue: true

receivers:
  - name: "slack"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxx'
        send_resolved: true
        channel: 'monitoring'
        text: "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

  - name: "pager"
    webhook_config:
      - url: http://a.b.c.d:8080/send/sms
        send_resolved: true
0
投票

我有具有以下配置的alertmanagerconfig，现在我需要将信息警报指向空接收器，我可以有多个接收器和接收器吗？

kind: AlertmanagerConfig
metadata:
  name: Prometheus-alertmanager-config
  namespace: Prometheus
spec:
  route:
    receiver: alert-email-pagerduty-config
    groupBy: ['alertname', 'priority','severity']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 15m
    continue: true
  receivers: 
  - name: alert-email-pagerduty-config
    emailConfigs:
    - to: {{.to_email}}
      sendResolved: true
      from: {{.from_email}}
      smarthost:  {{.smarthost}}
      authUsername: {{.mail_username}}
      authPassword:
        name: 'alert-smtp-password'
        key: 'password'
      requireTLS: true
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: alert-smtp-password
  namespace: prometheus
stringData:
  password: {{.mail_password}}
0
投票

注意！确保 

continue: true
和下一个接收者之间没有任何间隙或换行符。

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 30s

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "one"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "one"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium|Low|warning|info"
      continue: true
    - receiver: "two"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium"
      continue: true
    - receiver: "three"
      group_wait: 10s
      match_re:
        severity: "Critical|High"
      continue: true

receivers:
  - name: "one"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83CHL/B05G3RPUE03/kEFGLfmR0RtDVEsFUwrlkKv'
        send_resolved: true
        channel: 'one'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "two"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83kuL/B05GGJEAV8R/LEEkDn24LE7BAVQr4HNWW76h'
        send_resolved: true
        channel: 'two'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "three"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/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'
        send_resolved: true
        channel: 'three'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}
0
投票

我有这个配置，但警报仅在一个通道中发送，而不是 2/

    config:
      global:
        
        resolve_timeout: 5m
      route:
        group_by: ['alertname', 'job', 'severity']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: "default"
        routes:

          - match:
              alertname: PreprodMonitor
              severity: warning|info  
            receiver: "default"
            continue: true

          - match:
              alertname: PreprodMonitor
              severity: critical
            receiver: "critical" 
            continue: true

                      
      receivers:
         # send to a slack channel. This is being used by the warning critical
        - name: "critical"
          slack_configs:
            - send_resolved: true
              api_url: https://hooks.slack.com/services/critical
              channel: '#kube-preprod2-alerting-critical'
              text: "{{ range .Alerts }}{{ .Annotations.message }}\n{{ .Annotations.runbook_url }}\n{{ end }}" 
              icon_url: https://avatars3.githubusercontent.com/u/3380462
              title: "{{ .CommonLabels.alertname }} - severity: {{ .CommonLabels.severity }}\n"
              # title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
            # send to a slack channel. This is being used by the warning severity
        - name: "default"
          slack_configs:
            - send_resolved: true
              api_url: https://hooks.slack.com/services/warning
              channel: '#kube-preprod2-alerting'
              # title: "{{ range .Alerts }}{{ .Labels.alertname }} - {{ .Labels.severity }}\n{{ end }}"
              text: "{{ range .Alerts }}{{ .Annotations.message }}\n{{ .Annotations.runbook_url }}\n{{ end }}" 
              icon_url: https://avatars3.githubusercontent.com/u/3380462
              title: "{{ .CommonLabels.alertname }} - severity: {{ .CommonLabels.severity }}\n"
              # te
0
投票

这对我来说很适合你的例子，伙计们:)

##--Configuration v4.1 :: multiple receivers (Telegram + Gmail)
#
global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true

templates:
  - '/etc/alertmanager/_templates/*.tmpl'

route:
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 5m
  #
  ## default receiver (only one can be assigned)
  #receiver: blackhole
  receiver: gmail-notifications
  #receiver: tg-notifications
  #
  routes:
    - receiver: tg-notifications
      group_wait: 10s
      match:
        alertname: PrometheusTargetMissing
        severity: critical
      continue: true
    - receiver: gmail-notifications
      group_wait: 10s
      match:
        alertname: PrometheusTargetMissing
        severity: critical
      continue: true
  #
receivers:
  ## Sends Notifications to Null (discard)
  - name: blackhole
  #
  ## Sends Notifications to Telegram
  - name: tg-notifications
    telegram_configs:
    - send_resolved: true
      disable_notifications: false
      http_config:
        follow_redirects: true
        enable_http2: true
      api_url: https://api.telegram.org
      chat_id: <secret>
      bot_token_file: "/etc/alertmanager/_secrets/sec_tg_token.txt"
      #
      # Custom message-template
      parse_mode: HTML
      message: '{{ template "telegram.default" . }}'
  #
  ## Sends Notifications to Gmail
  - name: gmail-notifications
    email_configs:
    - send_resolved: true
      require_tls: true
      from: <hidden>
      to: <hidden>
      hello: localhost
      smarthost: smtp.gmail.com:587
      auth_username: <hidden>
      auth_identity: <hidden>
      auth_password_file: /etc/alertmanager/_secrets/sec_smtp_pwd
      # Custom message-template
      headers:
        From: <hidden>
        To: <hidden>
        Subject: '{{ template "custom_mail_subject" . }}'
      html: '{{ template "custom_mail_html" . }}'

普罗米修斯 v2.47.2 警报管理器 v0.26.0

我对这个 sythtax 有问题

路线： - 接收者：tg-通知 组等待：10秒 匹配项： 严重性：“严重|高” 继续：正确

就我而言，它不起作用，子路由也没有触发，但只有默认路由工作正常。我花了 3 天的时间阅读了大量的例子，但没有用 😭

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.