Add new alert that fires if etcd backup metrics are missing for 12h. (#…

…946) * Add new alert that fires if etcd backup metrics are missing for 12h. * Add new alert that fires if etcd backup metrics are missing for 12h. * Update etcdbackup.rules.yml
giantswarm · Nov 13, 2023 · 0002f27 · 0002f27
1 parent e1cd463
commit 0002f27
Show file tree

Hide file tree

Showing 2 changed files with 16 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Add new alert that fires if etcd backup metrics are missing for 12h.
+
 ## [2.139.0] - 2023-11-07
 
 ### Added

diff --git a/helm/prometheus-rules/templates/alerting-rules/etcdbackup.rules.yml b/helm/prometheus-rules/templates/alerting-rules/etcdbackup.rules.yml
@@ -68,3 +68,15 @@ spec:
         severity: page
         team: {{ include "providerTeam" . }}
         topic: etcd-backup
+    - alert: ETCDBackupMetricsMissing
+      annotations:
+        description: '{{`ETCD backup metrics are missing`}}'
+        opsrecipe: etcd-backup-metrics-missing/
+      expr: absent(etcd_backup_latest_attempt)
+      for: 12h
+      labels:
+        area: kaas
+        cancel_if_outside_working_hours: "true"
+        severity: page
+        team: {{ include "providerTeam" . }}
+        topic: etcd-backup