[ci.jenkins.io / BOM] Diagnose slowness when >200 parallel pipeline steps are running

[dduportal]

Service(s)

ci.jenkins.io

Summary

Follow up of jenkinsci/bom#1969 and #3521.

The BOM builds are taking way more time (and money) than they should given the parallelization and infrastructure capacity.
A lot of steps are way slower than expected when there are hundreds of parallel jobs running on ci.jenkins.io.

This issue is about identifying the most probable causes by ruling out the Kubernetes plugin and/or the AWS EKS cloud used for agents.

The following tasks are expected:

• PR to rule out (or point at) the Kubernetes plugin by trying a BOM build using Azure VMs. Also a nice stress test for [Sponsorships] Setup the secondary Azure subscription to consume the sponsor credits #3818
  • Extend quota on the new sponsorship subscription to support 200 parallel VM agents
  • Check if existing VM template are good for the BOM build (same size in CPU/memory as the containers currently used) or create a new on
  • Open a PR with labels and check behavior of the build
• Analyze result and report

Reproduction steps

No response

[dduportal]

Update: quota increase requested in the new subscription to allow for 1000 vCPUs to sustain this experiment

[jglick]

rule out (or point at) the Kubernetes plugin by trying a BOM build using Azure VMs

Fine enough if that is straightforward to do, but the most direct diagnosis would be to capture a controller thread dump at the moment that a particular (sh?) step is running without the agent actually running the corresponding external process, or that a step has supposedly completed yet the next step in the pipeline has not yet started. Being able to reproduce the problem and look at a thread dump could potentially mean immediately zeroing in one some 🤦 mistake in some plugin that is easily solved once we see it.

[dduportal]

The past months showed that moving the agent kubernetes cluster close to the controller (from AWS to Azure) did solve the problem. As such I'm closing the issue as not reproducible.