-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix promtail alerts for mimir #1046
Conversation
@@ -14,7 +13,7 @@ spec: | |||
annotations: | |||
description: '{{`Scraping of all promtail pods to check if one failed every 5 minutes.`}}' | |||
opsrecipe: promtail-is-not-running/ | |||
expr: count(up{app="promtail"} == 0) > 0 | |||
expr: count(up{container="promtail"} == 0) > 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is cluster_type="management_cluster"
missing ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No promtail should alert on all clusters
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even for customer promtail if they have one ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do manage them so yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But it's mostly about those in kube-system
If that pages too much then we can change that behavior @marieroque maybe splitting the promtail in kube-system and others, let;s see :) |
Before adding a new alerting rule into this repository you should consider creating an SLO rules instead.
SLO helps you both increase the quality of your monitoring and reduce the alert noise.
Towards: giantswarm/roadmap#3157
This PR fixes promtail alerts and make them work in mimir
Checklist
oncall-kaas-cloud
GitHub group).