[ruler] Support external labels

[bilbof]

Is your feature request related to a problem? Please describe.

It's not possible to set extra labels on series at the ruler, like it is with Prometheus with external_labels. So you can't set e.g. 'cluster' or 'environment' etc.

Describe the solution you'd like

Something like what Prometheus provides:

global:
  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    [ [<labelname>](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelname): [<labelvalue>](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelvalue) ... ]

[pracucci]

👋

So you can't set e.g. 'cluster' or 'environment' etc.

Could you share more about your use case, please? Why would you want to set, for example, "cluster" on the ruler series? Mimir typically is deployed as a centralised metrics database, collecting metrics from multiple clusters, so the "cluster" label is expected to be set in ingested metrics (received from Prometheus or the Agent) and then the ruler just use it.

My previous question follows the "cluster" example you made, but generally speaking we haven't seen the need of having external labels support in the Mimir ruler so far, so I would like to better understand your use case. Thanks!

[tgrondier]

Hi! @bilbof has not answered but I will share our use case:

We have a single dashboard setup as a receiver of all our alertmanager in different environments (testing, staging, production, etc). We rely on external labels sent by prometheus to alertmanager to separate and categorize alerts automatically in the dashboard. Without that, all alerts are grouped together as if they originated from the same environment

EDIT: We 'fixed' it by using mimirtool rules prepare

[jleloup]

A similar use cas I have that could benefit from having external labels set bu Mimir Ruler:

I am using Sloth to produce PrometheusRules resources which are then loaded into Mimir Ruler via Grafana Agent mimir.rules.kubernetes.
During this process I make sure to load these rules into all my Mimir tenants (which are representations of my environments producing metrics)

The problem then is that I am relying on a label environment in my alerts to be used later on for alert routing: think different Slack channels per environment for example.
Usually Prometheus external labels are enough to do the trick though in the context of these SLO rules:

• there is no possibility to add such external label within Mimir Ruler (neither in Grafana Agent during the provisioning of those rules into Mimir Ruler)
• Since I am not handling the rules directly -this is done by SLoth): I don't have the possibility to either add this label explictely or tweak the PromQL queries to keep it from the raw metrics

[QuentinBisson]

We have the same problem and the issue is that with prometheus we crafted a lot of alerts using aggregations (like sum by) and used external labels to add those labels in alertmanager but with Mimir, we would need to add those labels to all by clauses of all alerts which makes them then really hard to read when you have more than just 1 external label

[ying-jeanne]

We need to pick up that PR, adding some tests and fixing nits. This can be done by origin author or someone else who is willing to contribute.

[bilbof]

Could you share more about your use case, please? Why would you want to set, for example, "cluster" on the ruler series? Mimir typically is deployed as a centralised metrics database, collecting metrics from multiple clusters, so the "cluster" label is expected to be set in ingested metrics (received from Prometheus or the Agent) and then the ruler just use it.

Hey @pracucci, sorry for the delayed response. I'm no longer working on that project, but IIRC the use case is:

1. If you run an instance of Mimir per cluster/team/environment
2. And ship alerts to a centralized alert store
3. It would be useful to add labels to those alerts to filter by source/department/etc

If you want to give a dedicated instance of Mimir to a team/department/cluster/environment, and aggregate alerts, then global.external_labels would be useful. The alternative is having this logic pushed down to PrometheusRules/Prometheus instances which is hard to manage.