[BUG] Recreating existing pods when replicas changed

[davordbetter]

Describe the bug
At the moment I have 2 replicas

opensearch-cluster-master-0               1/1     Running   0          14m
opensearch-cluster-master-1               1/1     Running   0          16m

And if I changed in values.yaml
replicas: 2 -> 3
I get running master-2, but master-1 is terminated and after master-1 is ready again, master-0 gets terminated
opensearch-cluster-master-0 1/1 Running 0 15m
opensearch-cluster-master-1 1/1 Terminating 0 18m
opensearch-cluster-master-2 1/1 Running 0 36s

To Reproduce
Installed chart as subchart and added this to my values.yaml

opensearch:
  enabled: true
  clusterName: "opensearch-cluster"
  image:
    tag: 2.14.0
  replicas: 3
  persistence:
    storageClass: "azuredisk-csi-premium-retain"
    size: 20Gi
  envFrom:
    - secretRef:
        name: app-opensearch
    - configMapRef:
        name: app-opensearch
  config:
    opensearch.yml: |
      cluster.name: opensearch-cluster
      network.host: 0.0.0.0
  sysctlInit:
    enabled: true

Expected behavior
Not terminating existing masters

Host/Environment (please complete the following information):

• Helm Version: v3.14.2
• Kubernetes Version: 1.28.5

Additional logs:
opensearch-master-0.log

[prudhvigodithi]

[Triage]
Hey @davordbetter I assume the pods are restarted to reach the quorum. After sometime is the cluster stable after increasing the replicas?

[davordbetter]

Correct.

I tried kubectl scale and it worked as expected (scape up, only adding pod-2. scale down, only terminating pod-2).

While if I scale via helm chart.
pod-0 and pod-1 are alive and healty.
scale up to 3
pod-2 init and when it becomes healty, pod-1 is terminated and recreated. When pod-1 is healhty, same goes for pod-0