split ksm alerts in 2 separate ones

[QuantumEnigmaa]

This PR splits the KSMDown alert into 2 different alerts :

• the KSMDown one which triggers whenever the KSM component is actually down
• the KSMNotRetrievingMetrics one which triggers when KSM is not able to retrieve metrics from the cluster

Checklist

• Update CHANGELOG.md
• Add Unit tests
• Follow Alert structure
• Consider creating a dashboard (guidelines) (if it does not exist already) to help oncallers monitor the status of the issue.
• Request review from oncall area, as well as team (e.g: oncall-kaas-cloud GitHub group).

[QuentinBisson]

What's the difference with KubeSecretMetricMissing and KubeStateMetricsSlow?

[QuentinBisson]

Also we have a KSMdown inhibition :)

[QuantumEnigmaa]

From my own understanding, KubeStateMetricsSlow is monitoring the response time from KSM to make sure it doesn't take too long to retrieve metrics while KubeStateMetricsNotRetrievingMetrics is making sure that there are actually metrics retrieved by KSM

But maybe we can consider that if KSM is taking too long to retrieve metrics then it means that it's unable to retrieve it and thus we can get rid of KubeStateMetricsNotRetrievingMetrics to only rely on KubeStateMetricsSlow

[QuentinBisson]

Sure about the slow one but is this alert not the same as those

prometheus-rules/helm/prometheus-rules/templates/alerting-rules/kube-state-metrics.rules.yml

Line 31 in 737a5e9

- alert: KubeConfigMapCreatedMetricMissing

?

Also maybe we should regroup all KSM related alerts to this file

[QuantumEnigmaa]

Not sure, is kube_configmap_created a random metric normally retrieved by KSM that we check to make sure KSM is retrieving metrics in general ?

In that case I'd prefer also keeping the new alert because its name is more straightforward about its usage and moreover it indicates that KSM as a whole is not able to retrieve metrics while the KubeConfigMapCreatedMetricMissing would indicate that KSM is not able to retrive metrics from a particular instance

[QuentinBisson]

Well kube_configmap_created is the metric exposed by KSM when access the apiserver but we can have both if you prefer

[QuantumEnigmaa]

I guess it would make sense to have both yes

[AndiDog]

Does this inhibition make sense for KubeStateMetricsDown itself?

[hervenicol]

Inhibitions work with a source label and a target label: if an alert with the source label fires, alerts with the target label are inhibited.

Labels inhibit_xxx are source labels. This basically means "when this alert fires, please inhibit alerts that depend on kube_state_metrics_down.

[hervenicol]

LGTM

[whites11]

If atlas is happy I am happy