Alloy monitoring

[Rotfuks]

When one component from alloy fails, we should get an alert.

Related incident: we had an incident where alloy-rules failed to load rules on all installations, and got no alert (https://gigantic.slack.com/archives/C01176DKNP4/p1720628254293449)

Probably Alloy mixins provide all we need:

• servicemonitors to retrieve data
• dashboards for easy investigation
• sensible alerts

[QuentinBisson]

For mixins, please, do it the same way as the mimir and loki mixins and do not import them by hand

[QuentinBisson]

@TheoBrigitte I think you added mixins already right? We are probably only missing sensible alerts?

[QuentinBisson]

Current status:

• Added alerts for alloy components health and alloy logging pipeline Alloy monitoring prometheus-rules#1410 and opsrecipes https://github.com/giantswarm/giantswarm/pull/32008

Missing:

• Opsrecipes for the alloy components health alerts
• Alloy monitoring pipeline alerts and ops-recipes.

@giantswarm/team-atlas do we want to add alerts related to alloy clustering for this? Is it too soon?

[QuentinBisson]

Current status:

Awaiting reviews:

• Logging pipeline alerts: Alloy monitoring prometheus-rules#1410
• Logging pipeline ops-recipes: https://github.com/giantswarm/giantswarm/pull/32008
• Small improvements on alerts: Do some little clean up on grafana cloud and alloy alerts before adding the alloy-metrics related alerts prometheus-rules#1415
• Alloy metrics alerts add alerts for alloy-metrics prometheus-rules#1417
• monitoring pipeline ops-recipes: https://github.com/giantswarm/giantswarm/pull/32043

[QuentinBisson]

Blocked waiting for reviews

[QuentinBisson]

This is done, will need adjustsments