[BUG] Restarting a rollout with workloadRef crashes the operator pod

[jorgelon]

Describe the bug
There are 2 ways to create a rollout in argo rollouts:

• The first one is with a single resource "rollout". This includes both logics: the deployment spec and the rollout spec. With the new 1.1.0 reloader release, restarting this rollout works great (deleting the pods and not creating a new replicaset)

• The second one is with 2 resources. A rollout including the rollout spec and calling an existing deployment with spec.workloadRef. And the referenced deployment with the deployment spec.
  Restarting this rollout makes the controller crash.

To Reproduce
Change the secret that reloads the rollout deployed via workloadRef

Expected behavior
It works same ways as with a single rollout resource

Environment

• Operator Version: v1.1.0
• Kubernetes/OpenShift Version: 1.29.6

Log (via stern)
reloader-7df5bc9f58-phgjj reloader E0918 08:33:15.113227 1 runtime.go:79] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]:
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19f31e0?, 0xc001912c48})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?})
reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}})
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).processNextItem(0xc000432d80)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).runWorker(...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(*Controller).Run in goroutine 66
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259
reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0 [recovered]
reloader-7df5bc9f58-phgjj reloader panic: runtime error: index out of range [0] with length 0
reloader-7df5bc9f58-phgjj reloader
reloader-7df5bc9f58-phgjj reloader goroutine 12 [running]:
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc6380?})
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
reloader-7df5bc9f58-phgjj reloader panic({0x19f31e0?, 0xc001912c48?})
reloader-7df5bc9f58-phgjj reloader /usr/local/go/src/runtime/panic.go:914 +0x21f
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.getContainerUsingResource({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:382 +0x238
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.updatePodAnnotations({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:399 +0xec
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.invokeReloadStrategy({0x1c33320, 0x1c33308, 0x1c33328, 0x1c33310, 0x1c33318, 0x1c33380, 0x1c33330, {0x1b0f268, 0x7}}, {0x1da78c8, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:392 +0x12d
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.PerformAction({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:222 +0x16b0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.rollingUpgrade({{0x1dd0348, 0xc00090d1e0}, {0x1da5f00, 0x0}, {0x1da5f28, 0xc000c26c10}}, {{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:184 +0x128
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.doRollingUpgrade({{0xc000c37350, 0x18}, {0xc000986eb0, 0xa}, 0xc0006b4cf0, {0x1b33cca, 0x23}, {0x1b30b76, 0x21}, {0xc000b3e3f0, ...}, ...}, ...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/upgrade.go:173 +0xb0c
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/handler.ResourceUpdatedHandler.Handle({{0x1add080, 0xc000688140}, {0x1add080, 0xc001892b40}, {0xc00043c208, 0xc00043c220}, {0x7f056fc315e0, 0xc000437d00}})
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/handler/update.go:32 +0x265
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).processNextItem(0xc000432d80)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:246 +0xd0
reloader-7df5bc9f58-phgjj reloader github.com/stakater/Reloader/internal/pkg/controller.(*Controller).runWorker(...)
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:230
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000114000?, {0x1da1620, 0xc00019baa0}, 0x1, 0xc000534000)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x9bfa40?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
reloader-7df5bc9f58-phgjj reloader k8s.io/apimachinery/pkg/util/wait.Until(0xc00035b7d0?, 0x9b8525?, 0xc000254600?)
reloader-7df5bc9f58-phgjj reloader /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
reloader-7df5bc9f58-phgjj reloader created by github.com/stakater/Reloader/internal/pkg/controller.(*Controller).Run in goroutine 66
reloader-7df5bc9f58-phgjj reloader /workspace/internal/pkg/controller/controller.go:215 +0x259

[MuneebAijaz]

it seems the code was written for spec.template.spec.containers and not for spec.workloadRef. This would be an enhancement i believe.
PRs are welcome

[praddy26]

Hi @MuneebAijaz

I would like to work on this issue. Could you please assign it to me?

[mikly-te]

Seeing this issue as well -- any ideas when this can be fixed?

[wololowarrior]

@mikly-te I can attempt to fix this. But im kinda struggling to recreate the issue.
Could you share the manifest files if possible to have a minimum reproducible example?

[mikly-te]

@wololowarrior here's a stripped down version of the manifest files, although not sure if it's minimal or not.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    secret.reloader.stakater.com/reload: webapps
  name: webapps
spec:
  replicas: 6
  selector:
    matchLabels:
      app: webapps
  strategy:
    canary:
      canaryService: webapps-canary
      scaleDownDelaySeconds: 300
      stableService: webapps
      steps:
        - setCanaryScale:
            weight: 100
        - setWeight: 0
        - pause:
            duration: 5m
        - setWeight: 100
        - pause:
            duration: 1m
      trafficRouting:
        istio:
          virtualService:
            name: webapps
            routes:
              - primary
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapps

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapps
spec:
  replicas: 0
  selector:
    matchLabels:
      name: webapps
  template:
    metadata:
    spec:
      containers:
        - env:
            - name: CONFIG_APP_NAME
              value: webapps
          envFrom:
            - secretRef:
                name: webapps
          image: webapps
          name: application

[wololowarrior]

Thanks @mikly-te ! Appreciate it
I'm also using a similar deployment approach.
For me reloader isnt crashing, but it isnt restarting the pods as well. I guess that's what the issue is. Will try to fix this.

Is reloader crashing for you?

[jorgelon]

This makes the controller crash

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
  labels:
    app: deployment
spec:
  replicas: 0
  selector:
    matchLabels:
      app: deployment
  template:
    metadata:
      labels:
        app: deployment
    spec:
      containers:
      - image: docker.io/nginx:mainline-alpine3.20-slim
        name: nginx

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: deployment
  labels:
    app: deployment
  annotations:
    secret.reloader.stakater.com/reload: "deployment"
spec:
  selector:
    matchLabels:
      app: deployment
  workloadRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: deployment
  strategy:
    blueGreen:
      activeService: deployment
      autoPromotionEnabled: false
      activeMetadata:
        labels:
          role: active

apiVersion: v1
kind: Service
metadata:
  name: deployment
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http

apiVersion: v1
kind: Secret
metadata:
  name: deployment
stringData:
  VARIABLE: "whatever"
type: Opaque