[kube-prometheus-stack] msteams_configs not found in type config.plain #3724

SergkeiM · 2023-08-25T08:53:53Z

Describe the bug a clear and concise description of what the bug is.

After upgrading to 48.4.0 and trying to use msteams in alertmanager:

receivers:
      - name: 'msteams'
        msteams_configs:
          - webhook_url: '<url>'
            send_resolved: true
            title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'

I get this error:

sync "kube-prometheus-stack/kube-prometheus-stack-alertmanager" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors: line 32: field msteams_configs not found in type config.plain

What's your helm version?

v3.10.3

What's your kubectl version?

v1.23.6

Which chart?

kube-prometheus-stack

What's the chart version?

48.4.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack
--version 48.4.0
--namespace kube-prometheus-stack
--values values.yml

Anything else we need to know?

No response

The text was updated successfully, but these errors were encountered:

arukiidou · 2023-08-25T14:50:08Z

@SergkeiM

prometheus-operator supports v0.25.0 yet.
- Please wait for crd updates to support additional configurations.
- upstream upgrade is coming [bot] [main] Automated version update prometheus-operator/kube-prometheus#2207
- next release cut (v68.0.0)

dbluxo · 2023-08-28T13:06:36Z

Hey @SergkeiM 👋 ,
just out of curiosity and for inspiration, could you please share your "msteams.text" template?

SergkeiM · 2023-08-28T19:11:44Z

Hi @dbluxo

I have 2 templates for msteams this one I use in ceph alertmanager (HTML)

title: '{% raw %}{{ if eq .Status "firing" }}🚨 <b>FIRING</b> 🔥{{- else -}}🙌 <b>RESOLVED</b> 🍻{{- end -}}{% endraw %}'
text: '{% raw %}
<b>Alert:</b> <code>{{ .CommonLabels.alertname }}</code><br>
<b>Severity:</b> <code>{{ .CommonLabels.severity | title -}}</code><br>
{{- if (index .Alerts 0).Annotations.summary }}<b>Summary:</b> {{ (index .Alerts 0).Annotations.summary }}<br><br>{{- end }}
{{ range .Alerts }}
{{- if .Annotations.description }}{{ .Annotations.description }}<br><br>{{- end }}
{{- if .Annotations.message }}{{ .Annotations.message }}<br><br>{{- end }}
{{- end }}{% endraw %}'

And this one in Grafana and Prometheus

title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
text: '{{ template "msteams.text" . }}'

msteams.tmpl: |
      {{ define "msteams.text" -}}
      **Alert:** {{ .CommonLabels.alertname }}\
      **Service:** {{ .CommonLabels.service }}\
      **Severity:** {{ .CommonLabels.severity | title -}}\
      {{- if (index .Alerts 0).Annotations.summary }}
      **Summary:** {{ (index .Alerts 0).Annotations.summary }}
      {{- end }}
      {{ range .Alerts }}
      {{- if .Annotations.description }}{{ .Annotations.description }}\{{- end }}
      {{- if .Annotations.message }}{{ .Annotations.message }}\{{- end }}
      {{- end }}
      {{- end }}

For some reason HTML doesn't work in Prometheus (Alertmanager) so had to use markdown.

EraYaN · 2023-08-30T11:21:08Z

This is still a problem with v49 of the chart (with updated alertmanager etc).

SergkeiM · 2023-08-30T13:47:46Z

Hi @EraYaN

This is becuase prometheus is not yet updated the image: https://quay.io/repository/prometheus/prometheus?tab=tags&tag=latest

The last version kube-prometheus-stack uses is: 'v2.46.0', but msteams was added after, it is available in main though

helm-charts/charts/kube-prometheus-stack/values.yaml

Line 2897 in 2971286

tag: v2.46.0

So we need to wait until 'prometheus' image is released (v2.47.0), and after chart is updated.

But is defintley confusing, I think first all related things has to be updated, and after the charts, not vice-versa.

SergkeiM · 2023-09-11T10:24:38Z

Hi @arukiidou the above was fixed with release of 51.0.0, but there is a new error now:

failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 32: field msteams_configs not found in type alertmanager.receiver

zeritti · 2023-09-11T10:48:33Z

Support for alertmanager's msteams_configs did not make it to the latest release of Prometheus operator (0.68.0). The feature cannot be used with the operator as yet (is being worked on in PR 5778).

arukiidou · 2023-09-11T12:22:11Z

To make matters worse,
I worked on the prometheus bump alertmanager, but did not make it in v2.47.0. It will v2.48.0.

tadavid-cae · 2023-11-06T15:31:48Z

and now what?

zeritti · 2023-11-06T16:12:35Z

Support for the msteams receiver is present in the latest Prometheus operator release 0.69.0. You can try that out (with CRDs from the same release) or wait until the next chart's major release is out.

95gabor · 2023-11-06T17:39:56Z

With @tadavid-cae we found that the issue was fixed in: prometheus/alertmanager#3495, and it has not been released yet, but it is merged to the main branch.
To use the fixed version, you need to override the alert manager tag in your helm values files: (until a new version is released)

alertmanager:
  tag: main@sha256:1c3941499c0379e516b75fcfa28f991342e12e922cdf499620a7e4b33dd5060e

SergkeiM · 2023-11-06T18:53:07Z

Hi @95gabor I don't think is alertmanager issue, becuase I do use MSTeams in ceph with alertmanager 0.26, also works in grafana, the only problems is prometheus and we wait for 2.48

tadavid-cae · 2023-11-07T07:11:22Z

@SergkeiM Probably I am missing something, but how would prometheus be problematic here when sending alerts to teams is not working which is done by alertmanager?

bmgante · 2023-11-25T00:17:38Z

Can anyone confirm if latest version on kube-prometheus-stack already addresses this problem?

mihaico · 2023-11-27T10:12:03Z

I can confirm this is working with kube-prometheus-stack:54.1.0, the only comment is that one needs to specify both text and title due to a bug that was fixed post-release in prometheus-operator prometheus-operator/prometheus-operator#6109

here is an example:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: test-service
  namespace: test-service
  labels:
    app: alertmanager-config
spec:
  route:
    receiver: 'test-service'
    routes:
    - receiver: 'test-service'
      continue: true
      matchers:
      - name: 'namespace'
        matchType: '='
        value: 'test-service'
      - name: 'severity'
        matchType: '=~'
        value: 'warning|critical|WARNING|CRITICAL'
  receivers:
  - name: 'test-service'
    msteamsConfigs:
    - webhookUrl:
        key: webhook_url
        name: test-service-secret
      sendResolved: true
      title: '{{ template "msteams.default.title" . }}'
      text: '{{ template "msteams.default.text" . }}'

bmgante · 2023-11-27T10:27:14Z

@mihaico lastest chart version is 54.2.2, does that version still requires that workaround?

mihaico · 2023-11-27T10:30:34Z

@mihaico lastest chart version is 54.2.2, does that version still requires that workaround?

I see its using prometheus-operator:0.69.1 release, so I believe it still does need the workaround.

bmgante · 2023-11-27T11:20:55Z

Ok thanks for confirming @mihaico . I'll give it a try because I am currently using another deployment (prometheus-msteams) to be able to send notifications to teams via alertmanager and this default integration is way better 😉

SergkeiM · 2023-11-28T07:45:02Z

Confirm, in Helm chart version 54.2.2 works fine.

trallnag · 2023-12-05T08:14:53Z

Weird, not working for me yet.

Chart version 55.0.0
Prom operator version 0.70.0
Alertmanager verion 0.26.0

level=error ts=2023-12-05T07:52:38.675552699Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="sync "observability/kps-kube-prometheus-stack-alertmanager" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 26: field msteams_config not found in type config.plain\n line 32: field msteams_config not found in type config.plain\n line 38: field msteams_config not found in type config.plain\n line 44: field msteams_config not found in type config.plain\n line 50: field msteams_config not found in type config.plain\n line 56: field msteams_config not found in type config.plain\n line 62: field msteams_config not found in type config.plain\n line 68: field msteams_config not found in type config.plain\n line 74: field msteams_config not found in type config.plain\n line 80: field msteams_config not found in type config.plain"

SergkeiM · 2023-12-05T08:27:10Z

Hi @trallnag

They key is: msteams_configs not msteams_config

example:

receivers:
      - name: 'msteams'
        msteams_configs:
          - webhook_url: 'WEBHOOK_URLf'
            send_resolved: true
            title: '{{ if eq .Status "firing" }}🚨 [FIRING] 🔥{{- else -}}🙌 [RESOLVED] 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'

trallnag · 2023-12-05T08:33:24Z

Amazing, nice catch. Thanks!

borsenkOoleh · 2024-07-31T14:29:28Z

Hi @dbluxo

I have 2 templates for msteams this one I use in ceph alertmanager (HTML)

title: '{% raw %}{{ if eq .Status "firing" }}🚨 <b>FIRING</b> 🔥{{- else -}}🙌 <b>RESOLVED</b> 🍻{{- end -}}{% endraw %}'
text: '{% raw %}
<b>Alert:</b> <code>{{ .CommonLabels.alertname }}</code><br>
<b>Severity:</b> <code>{{ .CommonLabels.severity | title -}}</code><br>
{{- if (index .Alerts 0).Annotations.summary }}<b>Summary:</b> {{ (index .Alerts 0).Annotations.summary }}<br><br>{{- end }}
{{ range .Alerts }}
{{- if .Annotations.description }}{{ .Annotations.description }}<br><br>{{- end }}
{{- if .Annotations.message }}{{ .Annotations.message }}<br><br>{{- end }}
{{- end }}{% endraw %}'

And this one in Grafana and Prometheus

title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
text: '{{ template "msteams.text" . }}'

msteams.tmpl: |
      {{ define "msteams.text" -}}
      **Alert:** {{ .CommonLabels.alertname }}\
      **Service:** {{ .CommonLabels.service }}\
      **Severity:** {{ .CommonLabels.severity | title -}}\
      {{- if (index .Alerts 0).Annotations.summary }}
      **Summary:** {{ (index .Alerts 0).Annotations.summary }}
      {{- end }}
      {{ range .Alerts }}
      {{- if .Annotations.description }}{{ .Annotations.description }}\{{- end }}
      {{- if .Annotations.message }}{{ .Annotations.message }}\{{- end }}
      {{- end }}
      {{- end }}

For some reason HTML doesn't work in Prometheus (Alertmanager) so had to use markdown.

Hi @SergkeiM
Unfortunately, I can't find any documentation on how to add the custom template to kube-prometheus-stack values.yaml
Please explain to me how to do it. I have tried to do it with alertmanagerFiles, but it was not working

SergkeiM added the bug Something isn't working label Aug 25, 2023

SergkeiM mentioned this issue Aug 25, 2023

Add msteams prometheus/alertmanager#3324

Merged

arukiidou mentioned this issue Aug 25, 2023

[kube-prometheus-stack] Updates Alertmanager image to v0.26.0 #3721

Merged

3 tasks

SergkeiM mentioned this issue Nov 23, 2023

[kube-prometheus-stack] Missing msteamsConfigs in the alertmanagerconfigs crd #3933

Open

SergkeiM closed this as completed Nov 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kube-prometheus-stack] msteams_configs not found in type config.plain #3724

[kube-prometheus-stack] msteams_configs not found in type config.plain #3724

SergkeiM commented Aug 25, 2023

arukiidou commented Aug 25, 2023 •

edited

Loading

dbluxo commented Aug 28, 2023

SergkeiM commented Aug 28, 2023

EraYaN commented Aug 30, 2023

SergkeiM commented Aug 30, 2023

SergkeiM commented Sep 11, 2023

zeritti commented Sep 11, 2023

arukiidou commented Sep 11, 2023

tadavid-cae commented Nov 6, 2023

zeritti commented Nov 6, 2023

95gabor commented Nov 6, 2023

SergkeiM commented Nov 6, 2023

tadavid-cae commented Nov 7, 2023

bmgante commented Nov 25, 2023

mihaico commented Nov 27, 2023

bmgante commented Nov 27, 2023

mihaico commented Nov 27, 2023

bmgante commented Nov 27, 2023

SergkeiM commented Nov 28, 2023

trallnag commented Dec 5, 2023

SergkeiM commented Dec 5, 2023

trallnag commented Dec 5, 2023

borsenkOoleh commented Jul 31, 2024

[kube-prometheus-stack] msteams_configs not found in type config.plain #3724

[kube-prometheus-stack] msteams_configs not found in type config.plain #3724

Comments

SergkeiM commented Aug 25, 2023

Describe the bug a clear and concise description of what the bug is.

What's your helm version?

What's your kubectl version?

Which chart?

What's the chart version?

What happened?

What you expected to happen?

How to reproduce it?

Enter the changed values of values.yaml?

Enter the command that you execute and failing/misfunctioning.

Anything else we need to know?

arukiidou commented Aug 25, 2023 • edited Loading

dbluxo commented Aug 28, 2023

SergkeiM commented Aug 28, 2023

EraYaN commented Aug 30, 2023

SergkeiM commented Aug 30, 2023

SergkeiM commented Sep 11, 2023

zeritti commented Sep 11, 2023

arukiidou commented Sep 11, 2023

tadavid-cae commented Nov 6, 2023

zeritti commented Nov 6, 2023

95gabor commented Nov 6, 2023

SergkeiM commented Nov 6, 2023

tadavid-cae commented Nov 7, 2023

bmgante commented Nov 25, 2023

mihaico commented Nov 27, 2023

bmgante commented Nov 27, 2023

mihaico commented Nov 27, 2023

bmgante commented Nov 27, 2023

SergkeiM commented Nov 28, 2023

trallnag commented Dec 5, 2023

SergkeiM commented Dec 5, 2023

trallnag commented Dec 5, 2023

borsenkOoleh commented Jul 31, 2024

arukiidou commented Aug 25, 2023 •

edited

Loading