On Rocky9 kubernetes worker node, the haproxy external check script times out and doesn't start

[aki-kamada]

Report

I am unable to set up PerconaXtraDBCluster because the haproxy external check script times out.
I confirmed that this problem occurs on Kubernetes worker nodes of Rocky9. This problem doesn't occur on Kubernetes worker nodes of Ubuntu 20.04.
I seem to haproxy happen in busy loop. Because cpu resource is very increase.
Of course, when I run check_pxc.sh within the haproxy pod, I can confirm that it connects to the mysql cluster normally.

I found that this problem happen in Amazon EKS(amazon linux 2023 with kubernetes 1.29).
amazonlinux/amazon-linux-2023#693

Please let me know if we need more informations.

More about the problem

I attached haproxy log, cr.yaml, ps command log in haproxy container and manually run external-check-script log.
Explore-logs-2024-09-10 16_44_39.txt
ps-command-on-haproxy-pod.txt
test-cluster00-haproxy00.txt
manually-run-external-check-script.txt

Steps to reproduce

1. setup pxc operator (I confirmed 3 version "1.13.0", "1.14.0" and "1.15.0")
2. kubectl apply -f test-cluster00-haproxy00.yaml. (running pod on Rocky9 kubernetes worker node)

Versions

1. Kubernetes 1.29.6 (problem occurs in Rocky9 worker node. doesn't problem occurs in Ubuntu 20.04 worker node)
2. Operator 1.130, 1.14.0 and 1.15.0 (I confirmed 3 versions)
3. Database 1.130, 1.14.0 and 1.15.0 (I confirmed 3 versions)

Anything else?

No response

[aki-kamada]

I found that this issue occurs when the worker node is using a large amount of memory (over 300GB in my environment).
This issue has been confirmed to occur with haproxy and slapd.

Resolution:
If you change the value of LimitNOFILE in the containerd.service on the Kubernetes worker node from infinity to 65535, haproxy will start.

Thanks,
Aki