chore: update remaining CI pipelines to k8s v1.31.0

[mauri870]

Proposed commit message

This PR fixes some leftovers from #40856.

Related issues

• Relates https://github.com/elastic/observability-dev/issues/3718
• Relates Add support for kubernetes version 1.31.0 #40856

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mauri870? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

This is breaking k8s integration tests for some reason, looking at the build logs I can't spot any meaningful error messages. Leaving this as draft for now.

[mauri870]

@elastic/obs-cloudnative-monitoring This PR is failing when running integration tests, but I can't spot any error messages in the logs. Any idea how I can go about debugging this? I can only spot a single error:

Error: failed modules: kubernetes

[gizas]

@mauri870 wondering if this has to do with the fact that you are trying to merge it 9.0 version and we still have some issues with the version of the packages.

For the troubleshooting I would advise to go to your local folder and :

cd metricbeat 
mage goIntegTest

See also here some more info for how to run go tests for metricbeat

Can you reproduce the issue locally?

[mauri870]

Can you reproduce the issue locally?

Thanks! I tested this locally and it does not seem to fail for me. I pushed a debug log to test it on BK and I got the actual error:

Error: failed to run integration tests for module kubernetes:
1 error: kind setup failed: running "kind create cluster --name metricbeat-9-0-0-921fd909f7-snapshot --kubeconfig /opt/buildkite-agent/builds/bk-agent-prod-gcp-1729021037068466664/elastic/beats-metricbeat/metricbeat/build/kind/metricbeat-9-0-0-921fd909f7-snapshot/kubecfg --wait 300s --image kindest/node:v1.31.0" failed with exit code 126

https://buildkite.com/elastic/beats-metricbeat/builds/10481#019291a8-1c9d-4180-b768-80e832625d12/191-2135

Trying this command locally works just fine, I'm not sure what exact problem the builder is experiencing.

[mauri870]

@gizas Turns out something broke between kind v0.20.0 and v0.24.0 that is causing failures in BK.

[mauri870]

Raised #41257 so we can look into the failure in a separate issue.

[mauri870]

Enabled auto-merge.