Why is "prometheus" label required on PrometheusRules?

[conallprendergast]

When running in a cluster with PrometheusRules defined, the operator will error when the rules do not contain a prometheus label.
Why is this label required?

[conallprendergast]

Seems that the prometheus label is commonly used so that prometheus operator can select rules via serviceMonitorSelector:

It may be worth allowing PrometheusRules that do not have a prometheus label

[talal]

The operator also uses the prometheus label as the prefix for the PrometheusRule resources that contains the absence alert rules (see aggregation for more details).

For example, if there is/are PrometheusRule resource(s) in a namespace with the prometheus label with value kubernetes then the PrometheusRule resource created by the operator which contains the corresponding absence alert rules will be called kubernetes-absent-metric-alert-rules.

[conallprendergast]

Would it make sense to have an optional default prefix for created PrometheusRule resource, for PrometheusRule resources that do not have a prometheus label?
Or possibly just to name the created PrometheusRule absent-metric-alert-rules?

[majewsky]

FWIW, we will have to think about this topic anyway. We have a ticket in our internal backlog concerning alerts that have the thanos-ruler label instead of the prometheus label.

[conallprendergast]

You could have a flag called "aggregage-rule" or something, that takes a template string. eg by default "{.prometheusrule.metadata.labels.prometheus}-absent-metrics"

This could allow aggregating to one new PrometheusRule per discovered rule, (eg, if we specified "{.prometheusrule.metadata.name}-absent-metrics") rather than one per namespace.

Would also allow custom naming and remove dependencies on any particular labels.

[conallprendergast]

For your information, I have created a PR #153 that would suit our use-case. Feel free to add to it, change it etc

[talal]

@conallprendergast I merged your changes in a new PR and made some improvements to the implementation.