Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix promtail alerts for mimir #1046

Merged
merged 2 commits into from
Mar 5, 2024
Merged

Fix promtail alerts for mimir #1046

merged 2 commits into from
Mar 5, 2024

Conversation

QuentinBisson
Copy link
Contributor

Before adding a new alerting rule into this repository you should consider creating an SLO rules instead.
SLO helps you both increase the quality of your monitoring and reduce the alert noise.


Towards: giantswarm/roadmap#3157

This PR fixes promtail alerts and make them work in mimir

Checklist

@QuentinBisson QuentinBisson self-assigned this Mar 5, 2024
@QuentinBisson QuentinBisson requested a review from a team as a code owner March 5, 2024 10:53
@@ -14,7 +13,7 @@ spec:
annotations:
description: '{{`Scraping of all promtail pods to check if one failed every 5 minutes.`}}'
opsrecipe: promtail-is-not-running/
expr: count(up{app="promtail"} == 0) > 0
expr: count(up{container="promtail"} == 0) > 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cluster_type="management_cluster" missing ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No promtail should alert on all clusters

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even for customer promtail if they have one ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do manage them so yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's mostly about those in kube-system

@QuentinBisson
Copy link
Contributor Author

If that pages too much then we can change that behavior @marieroque maybe splitting the promtail in kube-system and others, let;s see :)

@QuentinBisson QuentinBisson merged commit 2945da3 into master Mar 5, 2024
5 checks passed
@QuentinBisson QuentinBisson deleted the fix-promtail-alert branch March 5, 2024 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants