Kube-proxy 1.31.3 on ubuntu linux kernel 5.15.0-127-generic fails

[erichevers]

After upgrading the OS today to the latest version and also upgraded RKE2 to 1.31.3 the kube-proxy pods keep crashing.
Downgrading the linux kernerl to 5.15.0.124 fixed the issue and another node with 5.15.0.126 didn't have the issue, so it seems like it is 5.15.0-127 specific

uname -srn
Linux nl1k8s097 5.15.0-127-generic

kube-proxy logs:
E1217 15:37:53.421766 1 proxier.go:734] "Error cleaning up nftables rules" err="could not find nftables binary: exec: "nft": executable file not found in $PATH"
E1217 15:37:53.421828 1 proxier.go:734] "Error cleaning up nftables rules" err="could not find nftables binary: exec: "nft": executable file not found in $PATH"
I1217 15:37:53.435690 1 server.go:677] "Successfully retrieved node IP(s)" IPs=["192.168.12.94"]
E1217 15:37:53.435974 1 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using --nodeport-addresses primary"
I1217 15:37:53.470747 1 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
I1217 15:37:53.470947 1 server_linux.go:169] "Using iptables Proxier"
I1217 15:37:53.472139 1 proxier.go:255] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses" ipFamily="IPv4"
I1217 15:37:53.472589 1 server.go:483] "Version info" version="v1.31.3+rke2r1"
I1217 15:37:53.472613 1 server.go:485] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1217 15:37:53.473836 1 config.go:199] "Starting service config controller"
I1217 15:37:53.473869 1 shared_informer.go:313] Waiting for caches to sync for service config
I1217 15:37:53.473973 1 config.go:105] "Starting endpoint slice config controller"
I1217 15:37:53.473985 1 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
I1217 15:37:53.474419 1 config.go:328] "Starting node config controller"
I1217 15:37:53.474431 1 shared_informer.go:313] Waiting for caches to sync for node config
I1217 15:37:53.574315 1 shared_informer.go:320] Caches are synced for endpoint slice config
I1217 15:37:53.574480 1 shared_informer.go:320] Caches are synced for node config
I1217 15:37:53.574362 1 shared_informer.go:320] Caches are synced for service config
E1217 15:37:53.709945 1 proxier.go:1564] "Failed to execute iptables-restore" err=<
exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"
Error occurred at line: 17
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.

ipFamily="IPv6"
I1217 15:37:53.710050 1 proxier.go:833] "Sync failed" ipFamily="IPv6" retryingTime="30s"
E1217 15:38:23.765125 1 proxier.go:1564] "Failed to execute iptables-restore" err=<
exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"
Error occurred at line: 17
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
ipFamily="IPv6"
I1217 15:38:23.765155 1 proxier.go:833] "Sync failed" ipFamily="IPv6" retryingTime="30s"

[ppomes]

Same issue for me.

[kgtw]

Same issue on v1.29.9-rke2r1 with 5.15.0-1073-aws kernel, 5.15.0-1072-aws is fine.

[phuongvanle]

Same issue for me. 5.15.0-127-generic v1.30.6-rke2r1-build20241023

[brandond]

Kernel bug, see discussion at: k3s-io/k3s#11175 (comment)

https://lore.kernel.org/all/[email protected]/

Upgrade or downgrade your kernel.

[erichevers]

FYI, Ubuntu kernel 5.15.0-128, which came out today still has this issue. I will be using version 126 which doesn't have this issue.

[paulrobu]

FYI, the kernel bug was fixed in the Linux package 5.15.0-130.140 released few hours ago, see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2091990

[erichevers]

Thanks for the quick fix. I've installed the 5.15.0-130.140 kernel on the DR k8s cluster nodes and can confirm this fixed the kube-proxy issue