Flake - ScyllaCluster [It] nodes are cleaned up after horizontal scaling

[zimnx]

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/logs/ci-scylla-operator-latest-e2e-gke-parallel/1838096007133401088

Snippet of what failed.

 ScyllaCluster [It] nodes are cleaned up after horizontal scaling
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_cleanup.go:27
  Timeline >>
  STEP: Creating a new namespace @ 09/23/24 06:12:18.989
  Sep 23 06:12:19.329: INFO: Created namespace "e2e-test-scyllacluster-fnm9v-0-khfqn".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-fnm9v-0-khfqn". @ 09/23/24 06:12:20.599
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-fnm9v-0-khfqn". @ 09/23/24 06:12:20.687
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-fnm9v-0-khfqn". @ 09/23/24 06:12:20.85
  STEP: Creating a single node ScyllaCluster @ 09/23/24 06:12:20.865
  STEP: Waiting for the ScyllaCluster to roll out (RV=5923) @ 09/23/24 06:12:20.989
  Sep 23 06:15:54.460: INFO: ScyllaCluster e2e-test-scyllacluster-fnm9v-0-khfqn/basic-2lxdd (RV=10854) is rolled out
  STEP: Verifying the ScyllaCluster @ 09/23/24 06:15:54.46
  Sep 23 06:15:54.489: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-fnm9v-0-khfqn"
  Sep 23 06:15:54.489: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-fnm9v-0-khfqn/basic-2lxdd"
  STEP: Validating no cleanup jobs were created @ 09/23/24 06:15:54.516
  STEP: Scaling the ScyllaCluster to 3 replicas @ 09/23/24 06:15:54.516
  STEP: Waiting for the ScyllaCluster to roll out (RV=10856) @ 09/23/24 06:15:54.655
  Sep 23 06:18:43.465: INFO: ScyllaCluster e2e-test-scyllacluster-fnm9v-0-khfqn/basic-2lxdd (RV=16532) is rolled out
  STEP: Verifying the ScyllaCluster @ 09/23/24 06:18:43.465
  Sep 23 06:18:43.576: INFO: Found 3 pvc(s) in namespace "e2e-test-scyllacluster-fnm9v-0-khfqn"
  Sep 23 06:18:43.576: INFO: Found 3 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-fnm9v-0-khfqn/basic-2lxdd"
  STEP: Validating cleanup jobs were created for all nodes except last one @ 09/23/24 06:18:43.661
  Sep 23 06:18:43.774: INFO: Current token ring hash is "COKCoKUpy4jZ5yd7XQMzuKU5GTtHq1ubTR5EsVjbylzTdBq4yf2McmX5mMONj62v7YyGnRPVsGqFxW5nd2uoCg=="
  [FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_cleanup.go:113 @ 09/23/24 06:18:43.774 
  [...]
     [FAILED] Expected
      <[]utils.ObserverEvent[*k8s.io/api/batch/v1.Job] | len:1, cap:5>: [...]
         to have length 2
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_cleanup.go:113 @ 09/23/24 06:18:43.774
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func4.1()
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_cleanup.go:113 +0xed3 

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out

/lifecycle rotten