RKE2 upgrade causes wrong deployment scaling for a short perod of time

[Torkolis]

Hi, I just upgraded my RKE2 single node cluster the usual way (running the script and restarting the server service).
I have a small postgresql deployment in there which uses a longhorn volume. I have explicitly defined replicas: 1 and strategy type recreate to avoid multiple postgresql instances trying to access the data files at the same time. But while this update was underway it seems that rke2 broke these rules and created multiple instances for a short period of time which of course corrupted the files so my database is not working anymore. While this is a test environment and the database is more or less empty, so I can just create a brand new volume, I still wonder if I am doing something wrong or if this should be concidered a bug or maybe some artifact because of the fact that there is only 1 node.

[brandond]

Have you opened an issue on the Longhorn side? I am not aware of any issues with RKE2 that would cause it to create additional Pods for ReplicaSets during an upgrade.

[Torkolis]

No I didn't open any issues yet. I'm not sure that longhorn has any fault in this, I will try to reproduce on a local VM without longhorn, but I would think, that it is kubernetes which restarts the pods and does scaling... Longhorn just provides storage.

[brandond]

All of the Longhorn components run as pods within the cluster. The Longhorn manager itself will create pods as necessary to meet the required replica count, for example. If you can't replicate the issue using normal Deployments then I would pursue this with Longhorn.

[Torkolis]

Do you mean that longhorn can create pods of my own (postgres) deployment?

[brandond]

No, it would not do that. Your post was unclear about what you were seeing multiple instances of - the workload, or the volume replica.

What does your workload deployment spec look like?

[Torkolis]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgresql
  namespace: {{ .Values.global.namespace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:14-alpine
        args: ["-c", "max_connections=1000"]
        ports:
        - containerPort: 5432
          name: listener
        env:
        - name: POSTGRES_PASSWORD
          value: admin
        - name: PGDATA
          value: /var/lib/postgresql/data/data
        volumeMounts:
        - name: datafiles
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: datafiles
        persistentVolumeClaim:
          claimName: datafiles-postgresql
      enableServiceLinks: false
      securityContext:
        fsGroup: 70
  strategy:
    type: Recreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datafiles-postgresql
  namespace: {{ .Values.global.namespace }}
spec:
  resources:
    requests:
      storage: 1Gi
  accessModes:
    - ReadWriteOnce
---
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  namespace: {{ .Values.global.namespace }}
spec:
  selector:
    app: postgresql
  ports:
  - name: listener
    port: 5432
    targetPort: 5432

[brandond]

There's a note about this in the docs: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#recreate-deployment

Note: This will only guarantee Pod termination previous to creation for upgrades. If you upgrade a Deployment, all Pods of the old revision will be terminated immediately. Successful removal is awaited before any Pod of the new revision is created. If you manually delete a Pod, the lifecycle is controlled by the ReplicaSet and the replacement will be created immediately (even if the old Pod is still in a Terminating state). If you need an "at most" guarantee for your Pods, you should consider using a StatefulSet.

It sounds like you need StatefulSet guarantees for your pods?

[Torkolis]

Alright, thanks a lot.