PXC Cluster cannot be created successfully

[drzombey]

Report

I tried to create a clean PXC cluster with the following cr. The joining process seems not to work; see logs. The pxc-0 pod is running successfully, but if the operator spawns the other pods, they cannot join successfully. I restarted the pxc-0, and the other PXC pods joined successfully. After some time, the pxc-1 and pxc-2 pods restarted and produced the same error log.

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: minimal-cluster
spec:
  crVersion: 1.16.1
  secretsName: minimal-cluster-secrets
  unsafeFlags:
    tls: true
    pxcSize: true
    proxySize: true
  upgradeOptions:
    apply: disabled
    schedule: "0 4 * * *"
  tls:
    enabled: false
  pxc:
    size: 3
    image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0
    autoRecovery: true
    nodeSelector:
      beta.kubernetes.io/arch: amd64
    tolerations:
      - key: "percona.com/pxc"
        operator: "Exists"
        effect: "NoSchedule"
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: gp3optimized
        resources:
          requests:
            storage: 500G
    resources:
      requests:
        memory: 32G
        cpu: "16"
    configuration: |
      [mysqld]
      innodb_buffer_pool_size=25G;
      sort_buffer_size=1048576
      join_buffer_size=4G
      innodb_read_io_threads=8
      innodb_log_file_size=3G
      innodb_log_buffer_size=128M
      innodb_buffer_pool_instances=25
      innodb_flush_log_at_trx_commit=1
      innodb_buffer_pool_instances=8
      innodb_file_per_table=1
      innodb_stats_on_metadata=0
      innodb_flush_method=O_DIRECT
      table_open_cache=1024
      table_definition_cache=996
      max_connections=500
      local_infile=1
      key_buffer_size=16M
      sql_mode=NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
      pxc_strict_mode = permissive
      wsrep_debug=CLIENT
      wsrep_provider_options="gcache.size=2G; gcache.recover=yes"
      binlog_format=ROW
      wsrep_provider_options="socket.checksum=1"
      wsrep_slave_threads=0

      [mysqld_safe]
      flush_cache = 1
      numa_interleave = 1

      [xtrabackup]
      compress=lz4
      parallel=8
      compress-threads=8

      [sst]
      xbstream-opts=--decompress      

  haproxy:
    enabled: false
    size: 3
    nodeSelector:
      beta.kubernetes.io/arch: amd64
    tolerations:
      - key: "percona.com/pxc"
        operator: "Exists"
        effect: "NoSchedule"
    image: perconalab/percona-xtradb-cluster-operator:main-haproxy

  proxysql:
    enabled: true
    size: 3
    image: perconalab/percona-xtradb-cluster-operator:main-proxysql
    resources:
      requests:
        memory: 1G
        cpu: 600m
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
    gracePeriod: 30
    nodeSelector:
      beta.kubernetes.io/arch: amd64
    tolerations:
      - key: "percona.com/pxc"
        operator: "Exists"
        effect: "NoSchedule"

  logcollector:
    enabled: false
    image: perconalab/percona-xtradb-cluster-operator:main-logcollector
  pmm:
    enabled: true
    image: perconalab/pmm-client:dev-latest
    serverHost: monitoring-service.pmm.svc.cluster.local

More about the problem

2024-12-27T16:21:23.327454Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2024-12-27T16:21:23.327489Z 0 [Note] [MY-000000] [Galera] Using CRC-32 (backward-compatible) for message checksums.
2024-12-27T16:21:23.327499Z 0 [Note] [MY-000000] [Galera] backend: asio
2024-12-27T16:21:23.327562Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2024-12-27T16:21:23.327780Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-12-27T16:21:23.327802Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-12-27T16:21:23.327906Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2024-12-27T16:21:23.332197Z 0 [Note] [MY-000000] [Galera] (9d1f08c1-afe4, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2024-12-27T16:21:23.332209Z 0 [Note] [MY-000000] [Galera] (9d1f08c1-afe4, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2024-12-27T16:21:23.332394Z 0 [Note] [MY-000000] [Galera] EVS version 1
2024-12-27T16:21:23.332434Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'minimal-cluster-pxc', peer '10.10.28.129:,minimal-cluster-pxc-0.minimal-cluster-pxc:'
2024-12-27T16:21:23.333290Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Connection refused
2024-12-27T16:21:23.333591Z 0 [Note] [MY-000000] [Galera] (9d1f08c1-afe4, 'tcp://0.0.0.0:4567') connection established to 552f5960-b4fb tcp://10.10.30.64:4567
2024-12-27T16:21:23.833640Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2024-12-27T16:21:23.833687Z 0 [Note] [MY-000000] [Galera] declaring 552f5960-b4fb at tcp://10.10.30.64:4567 stable
2024-12-27T16:21:23.833697Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2024-12-27T16:21:23.833825Z 0 [Note] [MY-000000] [Galera] Node 552f5960-b4fb state primary
2024-12-27T16:21:23.837209Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,552f5960-b4fb,31)
memb {
552f5960-b4fb,0
9d1f08c1-afe4,0
}
joined {
}
left {
}
partitioned {
}
)
2024-12-27T16:21:23.837221Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-12-27T16:21:23.842319Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://10.10.28.129:4567
2024-12-27T16:21:24.332851Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-12-27T16:21:24.332936Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-12-27T16:21:24.333034Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2024-12-27T16:21:24.333042Z 0 [Note] [MY-000000] [Galera] Opened channel 'minimal-cluster-pxc'
2024-12-27T16:21:24.333134Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2024-12-27T16:21:24.333155Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: Waiting for state UUID.
2024-12-27T16:21:24.333180Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 9d6cc1c5-c46e-11ef-83e1-9a9dbc61b0b8
2024-12-27T16:21:24.333197Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 9d6cc1c5-c46e-11ef-83e1-9a9dbc61b0b8 from 0 (minimal-cluster-pxc-0)
2024-12-27T16:21:24.333410Z 1 [Note] [MY-000000] [WSREP] wsrep running threads now: 2
2024-12-27T16:21:24.333414Z 2 [Note] [MY-000000] [WSREP] wsrep running threads now: 2
2024-12-27T16:21:24.333451Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2024-12-27T16:21:24.333567Z 2 [Note] [MY-000000] [Galera] open: enter(0,none,high priority,success,0,toi: -1,nbo: -1)
2024-12-27T16:21:24.333573Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 9d6cc1c5-c46e-11ef-83e1-9a9dbc61b0b8 from 1 (minimal-cluster-pxc-1)
2024-12-27T16:21:24.333583Z 2 [Note] [MY-000000] [Galera] open: leave(2,idle,high priority,success,0,toi: -1,nbo: -1)
2024-12-27T16:21:24.333590Z 2 [Note] [MY-000000] [Galera] before_command: enter(2,idle,high priority,success,0,toi: -1,nbo: -1)
2024-12-27T16:21:24.333591Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 13,
members = 1/2 (primary/total),
act_id = 50,
last_appl. = 36,
protocols = 4/11/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 439f8660-c46a-11ef-8485-7bdbb137cbbd
2024-12-27T16:21:24.333595Z 2 [Note] [MY-000000] [Galera] before_command: success(2,exec,high priority,success,0,toi: -1,nbo: -1)
2024-12-27T16:21:24.333600Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2024-12-27T16:21:24.333622Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [141, 141]
2024-12-27T16:21:24.333629Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> PRIMARY (TO: 51)
2024-12-27T16:21:24.333676Z 2 [Note] [MY-000000] [Galera] ####### processing CC 51, local, ordered
2024-12-27T16:21:24.333689Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from -1 upto current CC event 51 upto:-1
2024-12-27T16:21:24.333693Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 up to -1
2024-12-27T16:21:24.333700Z 2 [Note] [MY-000000] [Galera] Process first view: 439f8660-c46a-11ef-8485-7bdbb137cbbd my uuid: 9d1f08c1-c46e-11ef-afe4-73fad60b1326
2024-12-27T16:21:24.333709Z 2 [Note] [MY-000000] [Galera] Server minimal-cluster-pxc-1 connected to cluster at position 439f8660-c46a-11ef-8485-7bdbb137cbbd:51 with ID 9d1f08c1-c46e-11ef-afe4-73fad60b1326
2024-12-27T16:21:24.333715Z 2 [Note] [MY-000000] [Galera] server minimal-cluster-pxc-1 state change: disconnected -> connected
2024-12-27T16:21:24.333720Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
2024-12-27T16:21:24.333742Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-12-27T16:21:24.333764Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 9d1f08c1-c46e-11ef-afe4-73fad60b1326
2024-12-27T16:21:24.333771Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 11), state transfer needed: yes
2024-12-27T16:21:24.333840Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-12-27T16:21:24.333890Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-12-27T16:21:24.333899Z 2 [Note] [MY-000000] [Galera] State transfer required:
Group state: 439f8660-c46a-11ef-8485-7bdbb137cbbd:51
Local state: 00000000-0000-0000-0000-000000000000:-1
2024-12-27T16:21:24.333902Z 2 [Note] [MY-000000] [Galera] server minimal-cluster-pxc-1 state change: connected -> joiner
2024-12-27T16:21:24.333906Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
2024-12-27T16:21:24.333908Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-12-27T16:21:24.334041Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.10.24.115' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.39-30.1' --binlog 'binlog' )
2024-12-27T16:21:24.389298Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-12-27T16:21:24.389335Z 0 [ERROR] [MY-000000] [WSREP-SST] Missing version string in comparison
2024-12-27T16:21:24.389341Z 0 [ERROR] [MY-000000] [WSREP-SST] left-side: operation:< right-side:2.4.29
2024-12-27T16:21:24.389344Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* ERROR **********************
2024-12-27T16:21:24.563914Z 0 [Warning] [MY-000000] [WSREP-SST] Found a stale sst_in_progress file: /var/lib/mysql//sst_in_progress
2024-12-27T16:21:24.854213Z 2 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|10.10.24.115:4444/xtrabackup_sst//1
2024-12-27T16:21:24.854265Z 2 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
2024-12-27T16:21:24.854276Z 2 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: 439f8660-c46a-11ef-8485-7bdbb137cbbd
2024-12-27T16:21:24.854284Z 2 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 51, STRv: 3
2024-12-27T16:21:24.854350Z 2 [Note] [MY-000000] [Galera] IST receiver addr using tcp://10.10.24.115:4568
2024-12-27T16:21:24.854485Z 2 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-51, listening at: tcp://10.10.24.115:4568
2024-12-27T16:21:24.854860Z 0 [Note] [MY-000000] [Galera] Member 1.0 (minimal-cluster-pxc-1) requested state transfer from 'any'. Selected 0.0 (minimal-cluster-pxc-0)(SYNCED) as donor.
2024-12-27T16:21:24.854884Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 51)
2024-12-27T16:21:24.854911Z 2 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
2024-12-27T16:21:24.854919Z 2 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
2024-12-27T16:21:24.854925Z 2 [Note] [MY-000000] [Galera] GCache history reset: 439f8660-c46a-11ef-8485-7bdbb137cbbd:0 -> 439f8660-c46a-11ef-8485-7bdbb137cbbd:51
2024-12-27T16:21:25.175652Z 0 [Note] [MY-000000] [WSREP-SST] joiner: => Rate:[ 298 B/s] Avg:[ 298 B/s] Elapsed:0:00:00 Bytes: 167 B
2024-12-27T16:21:25.681596Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2024-12-27T16:21:25.749194Z 0 [Note] [MY-000000] [WSREP-SST] ............Waiting for SST streaming to complete!
2024-12-27T16:21:26.833070Z 0 [Note] [MY-000000] [Galera] (9d1f08c1-afe4, 'tcp://0.0.0.0:4567') turning message relay requesting off
2024-12-27T16:21:39.143260Z 0 [Note] [MY-000000] [WSREP-SST] joiner: => Rate:[0.00 B/s] Avg:[0.00 B/s] Elapsed:0:00:10 Bytes: 0.00 B
joiner: => Rate:[ 481KiB/s] Avg:[ 481KiB/s] Elapsed:0:00:13 Bytes: 6.33MiB
2024-12-27T16:21:39.148809Z 0 [Note] [MY-000000] [Galera] 0.0 (minimal-cluster-pxc-0): State transfer to 1.0 (minimal-cluster-pxc-1) complete.
2024-12-27T16:21:39.149017Z 0 [Note] [MY-000000] [Galera] Member 0.0 (minimal-cluster-pxc-0) synced with group.
2024-12-27T16:21:39.151206Z 0 [Note] [MY-000000] [WSREP-SST] Decompressing the backup at /var/lib/mysql//sst-xb-tmpdir
2024-12-27T16:21:39.268613Z 0 [Note] [MY-000000] [WSREP-SST] Preparing the backup at /var/lib/mysql//sst-xb-tmpdir
2024-12-27T16:21:39.278637Z 0 [Note] [MY-000000] [WSREP-SST] 2024-12-27T16:21:39.278570-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql --server-id=23725911 --innodb_flush_log_at_trx_commit=2 --innodb_flush_method=O_DIRECT --innodb_file_per_table=1 --innodb_buffer_pool_size=25G; --innodb_read_io_threads=8 --innodb_log_file_size=3G --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit=1 --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT
2024-12-27T16:21:39.278715Z 0 [Note] [MY-000000] [WSREP-SST] 2024-12-27T16:21:39.278707-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --socket=/tmp/mysql.sock --compress=lz4 --parallel=8 --compress-threads=8 --no-version-check=1 --use-memory=25G
2024-12-27T16:21:39.278721Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xtrabackup version 8.0.35-31 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 55ec21d7)
2024-12-27T16:21:39.280669Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2024-12-27T16:21:39.291335Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.10.24.115' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '1' --mysqld-version '8.0.39-30.1' --binlog 'binlog' : 1 (Operation not permitted)
2024-12-27T16:21:42.796650Z 0 [ERROR] [MY-000000] [WSREP] Failed to read uuid:seqno from joiner script.
2024-12-27T16:21:42.796691Z 0 [ERROR] [MY-000000] [WSREP] SST script aborted with error 1 (Operation not permitted)
2024-12-27T16:21:42.796759Z 3 [Note] [MY-000000] [Galera] Processing SST received
2024-12-27T16:21:42.796776Z 3 [Note] [MY-000000] [Galera] SST received: 00000000-0000-0000-0000-000000000000:-1
2024-12-27T16:21:42.796791Z 3 [System] [MY-000000] [WSREP] SST completed
2024-12-27T16:21:42.796857Z 2 [Note] [MY-000000] [Galera] str_proto_ver_: 3 sst_seqno_: -1 cc_seqno: 51 req->ist_len(): 67
2024-12-27T16:21:42.796919Z 2 [ERROR] [MY-000000] [Galera] Application received wrong state:
Received: 00000000-0000-0000-0000-000000000000
Required: 439f8660-c46a-11ef-8485-7bdbb137cbbd
2024-12-27T16:21:42.796926Z 2 [ERROR] [MY-000000] [Galera] Application state transfer failed. This is unrecoverable condition, restart required.
2024-12-27T16:21:42.796933Z 2 [Note] [MY-000000] [Galera] ReplicatorSMM::abort()
2024-12-27T16:21:42.796940Z 2 [Note] [MY-000000] [Galera] Closing send monitor...
2024-12-27T16:21:42.796946Z 2 [Note] [MY-000000] [Galera] Closed send monitor.
2024-12-27T16:21:42.796957Z 2 [Note] [MY-000000] [Galera] gcomm: terminating thread
2024-12-27T16:21:42.796970Z 2 [Note] [MY-000000] [Galera] gcomm: joining thread
2024-12-27T16:21:42.797094Z 2 [Note] [MY-000000] [Galera] gcomm: closing backend
2024-12-27T16:21:43.801178Z 2 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,552f5960-b4fb,31)
memb {
9d1f08c1-afe4,0
}
joined {
}
left {
}
partitioned {
552f5960-b4fb,0
}
)
2024-12-27T16:21:43.801240Z 2 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0
2024-12-27T16:21:43.801247Z 2 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view ((empty))
2024-12-27T16:21:43.801413Z 2 [Note] [MY-000000] [Galera] gcomm: closed
2024-12-27T16:21:43.801441Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2024-12-27T16:21:43.801517Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-12-27T16:21:43.801524Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2024-12-27T16:21:43.801529Z 0 [Note] [MY-000000] [Galera] Shifting JOINER -> OPEN (TO: 51)
2024-12-27T16:21:43.801536Z 0 [Note] [MY-000000] [Galera] New SELF-LEAVE.
2024-12-27T16:21:43.801548Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [0, 0]
2024-12-27T16:21:43.801553Z 0 [Note] [MY-000000] [Galera] Received SELF-LEAVE. Closing connection.
2024-12-27T16:21:43.801558Z 0 [Note] [MY-000000] [Galera] Shifting OPEN -> CLOSED (TO: 51)
2024-12-27T16:21:43.801564Z 0 [Note] [MY-000000] [Galera] RECV thread exiting 0: Success
2024-12-27T16:21:43.801659Z 2 [Note] [MY-000000] [Galera] recv_thread() joined.
2024-12-27T16:21:43.801669Z 2 [Note] [MY-000000] [Galera] Closing send queue.
2024-12-27T16:21:43.801673Z 2 [Note] [MY-000000] [Galera] Closing receive queue.
2024-12-27T16:21:43.801685Z 2 [Note] [MY-000000] [Galera] mysqld: Terminated.
2024-12-27T16:21:43.801690Z 2 [Note] [MY-000000] [WSREP] Initiating SST cancellation
2024-12-27T16:21:43Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=9b08babb531cc734c9d2b0e87fa9762b893fe484
Server Version: 8.0.39-30.1 Percona XtraDB Cluster (GPL), Release rel30, Revision 46271a0, WSREP version 26.1.4.3, wsrep_26.1.4.3

Thread pointer: 0x7f41ec000b60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f4202dd6bf0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x228bc01]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x128c7ef]
/usr/sbin/mysqld(handle_fatal_signal+0xd8) [0x128c8d8]
/lib64/libpthread.so.0(+0x12d10) [0x7f421d472d10]
/lib64/libc.so.6(abort+0x203) [0x7f421b7e1f41]
/usr/lib64/galera4/libgalera_smm.so(+0x462fb) [0x7f420eaf52fb]
/usr/lib64/galera4/libgalera_smm.so(+0x6b022) [0x7f420eb1a022]
/usr/lib64/galera4/libgalera_smm.so(+0x87dae) [0x7f420eb36dae]
/usr/lib64/galera4/libgalera_smm.so(+0x7afbd) [0x7f420eb29fbd]
/usr/lib64/galera4/libgalera_smm.so(+0x7becc) [0x7f420eb2aecc]
/usr/lib64/galera4/libgalera_smm.so(+0x7c16e) [0x7f420eb2b16e]
/usr/lib64/galera4/libgalera_smm.so(+0x9fe6d) [0x7f420eb4ee6d]
/usr/lib64/galera4/libgalera_smm.so(+0x7a2db) [0x7f420eb292db]
/usr/lib64/galera4/libgalera_smm.so(+0x4fb12) [0x7f420eafeb12]
/usr/sbin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0x12) [0x2c72912]
/usr/sbin/mysqld() [0x12def7d]
/usr/sbin/mysqld(start_wsrep_THD+0x39a) [0xf7d16a]
/usr/sbin/mysqld() [0x274a039]
/lib64/libpthread.so.0(+0x81ca) [0x7f421d4681ca]
/lib64/libc.so.6(clone+0x43) [0x7f421b7f98d3]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file using lib coredumper
PATH: (null)

stream closed EOF for core-staging/minimal-cluster-pxc-1 (pxc)

Steps to reproduce

1. apply cr
2. pxc-1 will be in CrashLoopBackOff
3. Restart pxc-0
4. Restart pxc-1 and pxc-2 will also spawn

Versions

1. Kubernetes EKS-1.30
2. Operator 1.16.1
3. Database 8.0

Anything else?

No response

[drzombey]

I was able to identify that the innodb_buffer_pool_size parameter produces this issue.