modprobe in hardened-kubernetes image is unusable due to missing shared libraries

[shalomjacob]

Environmental Info:
RKE2 Version: v1.28.15+rke2r1

Node(s) CPU architecture, OS, and Version: Ubuntu 22.04
Linux ip-172-31-6-165 5.15.0-1028-aws #32-Ubuntu SMP Mon Jan 9 12:28:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 Server node

Describe the bug:
trying to run kube-proxy in IPVS mode. However, we see the following warning in the kube-proxy logs:
time="2024-12-18T19:52:07Z" level=warning msg="Running modprobe ip_vs failed with message: modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory, error: exit status 127"

Steps To Reproduce:
Setup single node RKE2 cluster with Kubernetes v1.28.15+rke2r1. Set arguments for kube-proxy to run in ipvs.

kube-proxy-arg:
- proxy-mode=ipvs
- ipvs-scheduler=lc
- ipvs-strict-arp=true
kube-proxy-extra-mount:
- /usr/lib/:/usr/lib/

Once RKE2 starts up, kube-proxy logs have warning for missing libzstd.so.1 library.

kubectl logs kube-proxy-ip-172-31-6-165 -n kube-system
I1219 19:37:02.953843       1 node.go:141] Successfully retrieved node IP: 172.31.6.165
I1219 19:37:02.995881       1 server.go:632] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
time="2024-12-19T19:37:02Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory`, error: exit status 127"
I1219 19:37:03.003876       1 server_others.go:218] "Using ipvs Proxier"
I1219 19:37:03.003907       1 server_others.go:421] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR for family" ipFamily="IPv6"
I1219 19:37:03.003913       1 server_others.go:438] "Defaulting to no-op detect-local"
I1219 19:37:03.004267       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-LOAD-BALANCER-SOURCE-CIDR" truncatedName="KUBE-6-LOAD-BALANCER-SOURCE-CID"
I1219 19:37:03.004281       1 ipset.go:116] "Ipset name truncated" ipSetName="KUBE-6-NODE-PORT-LOCAL-SCTP-HASH" truncatedName="KUBE-6-NODE-PORT-LOCAL-SCTP-HAS"
I1219 19:37:03.004305       1 server.go:846] "Version info" version="v1.28.15+rke2r1"
I1219 19:37:03.004312       1 server.go:848] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1219 19:37:03.008236       1 config.go:188] "Starting service config controller"
I1219 19:37:03.008267       1 shared_informer.go:311] Waiting for caches to sync for service config
I1219 19:37:03.008298       1 config.go:97] "Starting endpoint slice config controller"
I1219 19:37:03.008307       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I1219 19:37:03.008314       1 config.go:315] "Starting node config controller"
I1219 19:37:03.008325       1 shared_informer.go:311] Waiting for caches to sync for node config
I1219 19:37:03.109390       1 shared_informer.go:318] Caches are synced for node config
I1219 19:37:03.109552       1 shared_informer.go:318] Caches are synced for endpoint slice config
I1219 19:37:03.109568       1 shared_informer.go:318] Caches are synced for service config
I1219 20:43:03.042786       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.248.141:443/TCP/10.42.0.74:444"
I1219 20:43:03.043096       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.248.141:443/TCP/10.42.0.75:444"
I1219 20:49:03.047448       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.95.181:8080/TCP/10.42.0.109:8080"
I1219 20:51:03.049130       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.181.243:8080/TCP/10.42.0.113:8080"
I1219 20:51:03.049182       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.13.77:8080/TCP/10.42.0.111:8080"
I1219 20:51:03.049223       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.203.157:9090/TCP/10.42.0.119:8081"
I1219 20:51:03.049276       1 graceful_termination.go:102] "Removed real server from graceful delete real server list" realServer="10.43.229.178:8080/TCP/10.42.0.114:8080"

Expected behavior:
No warning in the kube-proxy logs for missing libzstd.so.1 library

Actual behavior:
Warning message exists.

Additional context / logs:
The kube-proxy logs the warning but does not crash and stays running.

[brandond]

We’re copying over modprobe without copying over any of the shared libraries it needs.
https://github.com/rancher/image-build-kubernetes/blob/v1.30.6-rke2r1-build20241023/Dockerfile#L84-L88

brandond@dev01:~$ docker run --rm -it docker.io/rancher/hardened-kubernetes:v1.30.6-rke2r1-build20241023 modprobe
modprobe: error while loading shared libraries: libzstd.so.1: cannot open shared object file: No such file or directory

We need to be more careful about copying things out of BCI, they're not statically linked.

[brandond]

cc @rafaelbreno @brooksn