Setup Mimir on CAPA installation

[TheoBrigitte]

The goal here is to replace the current monitoring on a testing CAPA installation and have Mimir in place to send alerts and visualize data via Grafana.

This solution should be fully automated, so we can replicate this setup on multiple installations by using a toggle flag which would trigger the deployment and configuration of this monitoring solution. Automation for the new monitoring setup should no be added into the existing prometheus-meta-operator.

Here are some goals to reach :

• Being able to send alerts to OpsGenie from rules evaluated by Mimir
• Being able to link alerts and visualize actual data and dashboards in Grafana
• Being able to silence alerts (ideally using our silence repository)

Mimir Setup

Preview Give feedback

1. Create monitoring toggle flag #3087
   team/atlas topic/monitoring +2
   
2. Create default Mimir configuration #3088
   team/atlas topic/monitoring +2
   
3. Create object storage for Mimir #3089
   team/atlas topic/monitoring +2
   
4. Ingest data within Mimir #3152
   team/atlas +1
   
5. Add Mimir as datasource in Grafana #3158
   team/atlas +1
   
6. Setup Mimir's monitoring #3162
   component/mimir team/atlas +2
   
7. Setup PrometheusRules within Mimir ruler #3127
   component/mimir team/atlas topic/monitoring +3
   
8. Ensure alerting and recording rules are correctly evaluated by the mimir ruler #3157
   component/mimir team/atlas +2
   
9. Ensure cluster metrics are received #3163
   component/mimir +1
10. Disable rule evaluation in Prometheus if monitoring.enabled=true #3298
    team/atlas +1
    
11. Send Mimir's alert to OpsGenie #3160
    team/atlas +1
    
12. Send Mimir data to GrafanaCloud #3159
    3 of 3
    team/atlas +1
    
13. Deploy grafana-agent for PrometheusRules and monitor it #3301
    3 of 3
    team/atlas +1
    
14. https://github.com/giantswarm/giantswarm/issues/30281
15. Evaluate resources usage and cost for monitoring #3161
    team/atlas topic/monitoring +2
    
16. https://github.com/giantswarm/giantswarm/issues/30090
17. CI Checks for Mimir Alerts #3377
    team/atlas +1
    
18. https://github.com/giantswarm/giantswarm/issues/30218
19. Mimir HPA/autoscaling #3379
    blocked-upstream team/atlas +2
    
20. Send remote write data from agents to Mimir directly (short-circuiting MC prometheus) #3217
    component/mimir team/atlas topic/monitoring +3
    
21. https://github.com/giantswarm/giantswarm/issues/30310
22. https://github.com/giantswarm/giantswarm/issues/30836
23. https://github.com/giantswarm/giantswarm/issues/30834
24. https://github.com/giantswarm/giantswarm/issues/30835
25. https://github.com/giantswarm/giantswarm/issues/30837
26. https://github.com/giantswarm/giantswarm/issues/30887
27. https://github.com/giantswarm/giantswarm/issues/30833
28. https://github.com/giantswarm/giantswarm/issues/30758
29. https://github.com/giantswarm/giantswarm/issues/30963

After Mimir is succesfully setup on one installation, we can start getting our internal customers onboarded and check the new system before rolling it out everywhere and cleaning up the old system.

Migration and Cleanup

Preview Give feedback

1. https://github.com/giantswarm/giantswarm/issues/30365
2. https://github.com/giantswarm/giantswarm/issues/30982
3. Get rid of AlertManager ingress #3510
   component/alertmanager team/atlas +2
   
4. Deploy Mimir on (nearly) all CAPA installations #3090
   team/atlas topic/monitoring +2
   
5. https://github.com/giantswarm/giantswarm/issues/30839
6. Turtles general alerting review and migration to mimir #3315
   capi/migration component/mimir epic/capi team/turtles topic/observability +5
7. Phoenix general alerting review and migration to mimir #3314
   capi/migration component/mimir epic/capi team/phoenix topic/observability +5
8. https://github.com/giantswarm/giantswarm/issues/31094
9. Rollout of Mimir to goat and galaxy #3505
   team/atlas +1
   

   
10. Get rid of legacy prometheus on the CAPA MC #3218
    8 of 8
    blocked component/mimir component/prometheus team/atlas topic/observability +5
    
11. Mimir disabled configuration #3513
    blocked team/atlas +2
    
12. https://github.com/giantswarm/giantswarm/issues/31111

[QuentinBisson]

To unblock Mimir:

• Release announcement for mimir changes and opsctl open removal
• Adding missing datasource to all dashboards and finish atlas alert reviews
• Clean up the heartbeats per cluster and clean up prometheus heartbeat alert and remove the heartbeat creation
• Add information about the grafana agent in the heartbeat ops-recipe
• Prometheus to grafana cloud prometheus needs an alert and an ops-recipe update
• Explain in v20 release note that the prometheus agent resource usage will grow bigger
• Finish removing the prometheus on the MC to reduce resource usage
• Add HPAs on components we can but the ingester and add an ops-recipe on how to scale it up and down and also that we are alerted when we need to scale it up or down
• Review opsrecipes around prometheus/mimir
• Try out Mimir on Goat (private MCs)
• Remove the mimir specific opsgenie policies

[QuentinBisson]

cc @Rotfuks as we still need those items :)

[Rotfuks]

Here's the list of stories discussed from our onsite:

• giantswarm/giantswarm#30836
• giantswarm/giantswarm#30833
• giantswarm/giantswarm#30834
• giantswarm/giantswarm#30835
• giantswarm/giantswarm#30837
• giantswarm/giantswarm#30839

Need to do later:
Once we have alert review finished:

• Remove the mimir specific opsgenie policies that silence alerts coming from mimir out of BH

Once we have mimir rolled out:

• Clean up the heartbeats per cluster and clean up prometheus heartbeat alert and remove the heartbeat creation

[Rotfuks]

This is done, hurray! Great job @giantswarm/team-atlas !!!! 🎊