Update k3s for loadbalancer improvements

[brandond]

RKE2 tracking issue for:

• Embedded load-balancer behavior is flakey and hard to understand k3s-io/k3s#11334
• etcd-only and agent nodes do not properly fail back during apiserver outage k3s-io/k3s#11349

[VestigeJ]

See steps from k3s here k3s-io/k3s#11334 (comment)

[VestigeJ]

##Environment Details
Validated using VERSION=v1.30.8-rc1+rke2r1

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME               STATUS   ROLES                  AGE    VERSION
ip-node-11         Ready    etcd                   4h2m   v1.30.8+rke2r1
ip-node-136        Ready    etcd                   4h2m   v1.30.8+rke2r1
ip-node-62         Ready    <none>                 4h     v1.30.8+rke2r1
ip-node-22         Ready    control-plane,master   4h2m   v1.30.8+rke2r1
ip-node-164        Ready    etcd                   4h2m   v1.30.8+rke2r1

Config.yaml:

node-external-ip: node.22
server: https://node.164:9345
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cni: multus,cilium
embedded-registry: true
node-ip: node.22
disable-etcd: true

Validation

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.30.8-rc1+rke2r1
$ sudo INSTALL_RKE2_VERSION=$VERSION INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ go_rke2
$ sudo systemctl restart rke2-server
$ sudo reboot
$ set_kubefig rke2
$ PRODUCT=rke2
$ kgn
$ sudo systemctl stop rke2-server
$ sudo systemctl restart rke2-server
$ k create -f ingress.yaml
$ kg ing -A
$ curl -H 'Host: test1.com' http://node.53/name.html
$ kg ing -A -o wide
$ curl -H 'Host: test1.com' http://node.62/name.html
$ curl -H 'Host: test1.com' http://node.22/name.html
$ kgn
$ k delete node ip-node-53
$ kgn
$ curl -H 'Host: test1.com' http://node.62/name.html
$ kg ing -A
$ curl -H 'Host: test1.com' http://node.22/name.html

Results:

$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker1/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://worker2/name.html
othertest-deploy-dths2

$ k delete node.53
$ sudo journalctl -u rke2-agent -f

level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Removing server from load balancer rke2-api-server-agent-load-balancer: node.53:6443"
level=info msg="Updated load balancer rke2-api-server-agent-load-balancer server addresses -> [node.22:6443] [default: node.164:6443]"
level=info msg="Removing server from load balancer rke2-agent-load-balancer: node.53:9345"
level=info msg="Updated load balancer rke2-agent-load-balancer server addresses -> [node.22:9345] [default: node.164:9345]"

$ kgn

NAME               STATUS   ROLES                  AGE     VERSION
ip-node-11         Ready    etcd                   3h50m   v1.30.8+rke2r1
ip-node-136        Ready    etcd                   3h50m   v1.30.8+rke2r1
ip-node-62         Ready    <none>                 3h48m   v1.30.8+rke2r1
ip-node-22         Ready    control-plane,master   3h51m   v1.30.8+rke2r1
ip-node-164        Ready    etcd                   3h50m   v1.30.8+rke2r1

$ kg ing -A

NAMESPACE   NAME                CLASS   HOSTS       ADDRESS           PORTS   AGE
default     othertest-ingress   nginx   test1.com   node.62,node.22   80      27m

$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.62/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-6wjzs
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2
$ curl -H 'Host: test1.com' http://node.22/name.html
othertest-deploy-dths2

//after shutting down all control-plane nodes logs from the agent node
$ sudo journalctl -u rke2-agent -f

> sudo journalctl -u rke2-agent -f
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.22:9345: connect: connection refused" url="wss://node.22:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.53:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.53:9345: connect: connection refused"
level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.53:9345: connect: connection refused" url="wss://node.53:9345/v1-rke2/connect"
level=info msg="Connecting to proxy" url="wss://node.22:9345/v1-rke2/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.22:9345: connect: connection refused"

//After restarting nodes

rke2[1523]: time="2024-12-17T19:33:43Z" level=info msg="Getting list of apiserver endpoints from server"
rke2[1523]: time="2024-12-17T19:33:43Z" level=info msg="Server node.53:6443@UNCHECKED->RECOVERING from successful dial"
rke2[1523]: time="2024-12-17T19:33:48Z" level=info msg="Server node.53:6443@RECOVERING->FAILED from failed dial"
rke2[1523]: time="2024-12-17T19:33:48Z" level=info msg="Server node.22:6443@UNCHECKED->RECOVERING from successful dial"