feat: Upgrade monitoring stack

[gtirloni]

Loki 2.7.3 wasn't playing well with some queries from Grafana 10.3 and updating it to 2.9.6 fixed the issues.

While at it, I've updated Vector and Grafana/Prometheus as well.

Tested it in a live environment and also AIO. No new errors noticed in the logs.

Vector is a bit more noisy about issues but it doesn't prevent logs from being forwarded (especially, it complains when a log event is missing expected labels.. whereas before it would just ignore it).

[mnaser]

This is awesome, but I'd like this to be a test on how we do cherry-picks, since I'd like to have this in stable/zed too since monitoring stack updates are non-user affecting.

[mnaser]

/cherry-pick stable/zed

[vexxhost-bot]

@mnaser: once the present PR merges, I will cherry-pick it on top of stable/zed in a new PR and assign it to you.

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mnaser]

/cherry-pick stable/zed

[vexxhost-bot]

@mnaser: once the present PR merges, I will cherry-pick it on top of stable/zed in a new PR and assign it to you.

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mnaser]

régate

[mnaser]

regate

[vexxhost-bot]

@mnaser: #1071 failed to apply on top of branch "stable/zed":

Applying: feat(monitoring): Update to kube-prometheus-stack 58.0.0
Using index info to reconstruct a base tree...
M	roles/defaults/vars/main.yml
Falling back to patching base and 3-way merge...
Auto-merging roles/defaults/vars/main.yml
CONFLICT (content): Merge conflict in roles/defaults/vars/main.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 feat(monitoring): Update to kube-prometheus-stack 58.0.0
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mnaser]

/cherry-pick stable/zed

[vexxhost-bot]

@mnaser: #1071 failed to apply on top of branch "stable/zed":

Applying: feat(monitoring): Update to kube-prometheus-stack 58.0.0
Using index info to reconstruct a base tree...
M	roles/defaults/vars/main.yml
Falling back to patching base and 3-way merge...
Auto-merging roles/defaults/vars/main.yml
CONFLICT (content): Merge conflict in roles/defaults/vars/main.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 feat(monitoring): Update to kube-prometheus-stack 58.0.0
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vexxhost-bot]

@mnaser: new issue created for failed cherrypick: #1075

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.