Setup logging on CAPZ installations

[TheoBrigitte]

Towards: #311

In order to expand our logging infrastructure coverage we should extend support to CAPZ installations.

As for object storage Loki does support Azure blob storage.

Tasks

Preview Give feedback

1. Add Azure object storage (storage + role) #2967
   epic/logging provider/cluster-api-azure team/atlas +3
   
2. Configure Loki to use Azure object storage
3. Deploy logging on CAPZ installations
4. Enable logging on giantswarm CAPZ installations
5. Enable logging on customer CAPZ installations

[hervenicol]

🟢 Loki config should be ready (https://github.com/giantswarm/config/pull/2114)
🟢 it has been tested with a dev version of object-storage-operator (giantswarm/object-storage-operator#46)
🟠 unfortunately we don't receive promtail logs, looks like it can't resolve the ingress address

[marieroque]

My tests on glippy:

Manual steps to do first:

• enable logging on loki-app (loki-user-values configmap)
• enable logging on logging-operator (logging-operator-user-values configmap)

Then I encountered an issue with loki ingress. Loki DNS is not created.
Current issue on CAPZ:
https://github.com/giantswarm/giantswarm/issues/28848

After annotating loki-gateway ingress with giantswarm.io/external-dns: managed, loki DNS has been created and promtail on WC could sent data to loki:

[marieroque]

Logging enabled on CAPZ giantswarm testing installations only:

• gaggle: actually disabled because not enough resources (related to this issue https://github.com/giantswarm/giantswarm/issues/29213)
• glippy ✅
• goose: impossible to log into (ephemeral installation!?!)

[marieroque]

To have a grafana-agent working well on CAPZ WC, we need to edit the daemon set and change the listen-port from 80 to 8080.
That issue is fixed here https://github.com/giantswarm/giantswarm/issues/29368

=> It will be fixed in the new observability-bundle release.

[QuentinBisson]

Is it now fixed @marieroque ? Do we have logging on capz?

[QuentinBisson]

https://gigantic.slack.com/archives/C01176DKNP4/p1704804603915389 Might be worth checking

[marieroque]

✅ Tested on a WC on glippy

[marieroque]

🚀 Logging is working on CAPZ with:

• object-storage-operator 0.4.3
• logging-operator 0.3.1
• observability-bundle 1.0.0

⚠ Before enable logging on customer side (Continental/Wepa), wait for customer WC with o11y-bundle 1.0.0 as it contains a fix on grafana-agent

[marieroque]

Discussion about CAPZ upgrade: https://gigantic.slack.com/archives/C02HLSDH3DZ/p1705400551622879

[QuentinBisson]

Putting it back to backlog

[QuentinBisson]

All capz MCs run 1.25, let's en able logging on all capz mcs

[QuentinBisson]

Logging is enabled on capz