Update Buildkite version on Tartarus and Sverdrup

[ali-ramadhan]

Maybe this has already been extensively discussed but the GPU test suites on Buildkite fail often, requiring manual intervention to restart them for each PR.

The obvious solution is a bigger machine for testing, but I have two suggestions that are much easier to implement:

1. Updating Buildkite. Newer versions may be more stable. The latest version is 3.79 but Sverdrup is on v3.24.0 (almost 4 years old) and Tartarus is on v3.50.4.
2. If builds are failing due to too much resource competition, reducing the number of Buildkite agents on Sverdrup may help. Right now there are 16. I wonder if GPU builds will be more stable with 8-12. Some builds may be slower but if no one has to restart a test suite then that would make for a better developer experience.

[glwagner]

We think there is a race condition in the CI. Partly discussed on #3661 and also #3662, although one conclusions is that we should update to use the buildkite plugin (started on #3042)

[ali-ramadhan]

Makes sense. I saw enough different errors that I couldn't pinpoint a particular issue. Feel free to close this issue if it's a duplicate of #3661.

[navidcy]

I see your struggles @ali-ramadhan! I've been struggling with this... Usually just restart the build kite and wish for the best...

[navidcy]

(often the CPU build kite acts up...)