[release-1.31] loadbalancer improvements

[brandond]

Backport to release-1.31:

• Embedded load-balancer behavior is flakey and hard to understand #11334
• etcd-only and agent nodes do not properly fail back during apiserver outage #11349

[ShylajaDevadiga]

Tests to cover
#11334 (comment)

[fmoral2]

#11334 (comment)

-$   v1.31.3+k3s-b62f4747
 

Restart and validate cluster suite - ok 
https://github.com/rancher/distros-test-framework/blob/main/entrypoint/restartservice/restartservice_test.go

Restart CP nodes in reverse order - ok 

Stop and start one node  - ok 

reboot all + validate cluster  - ok 
https://github.com/rancher/distros-test-framework/blob/main/entrypoint/rebootinstances/rebootinstances_test.go

Delete 2 nodes and add 1 - ok 

Stop 2 nodes and check logs in the agent node - ok 

Dec 12 13:55:26 ip s[1819]: time="2024-12-12T13:55:26Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer:  
Dec 12 13:55:26   k3s[1819]: time="2024-12-12T13:55:26Z" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [18.191.129.77:6443 3.133.122.171:6443] [default: 1 :6443]"
Dec 12 13:55:26 i  k3s[1819]: time="2024-12-12T13:55:26Z" level=info msg="Started tunnel to  "
Dec 12 13:55:26 ip-  k3s[1819]: time="2024-12-12T13:55:26Z" level=info msg="Connecting to proxy" url="wss://18.191.129.77:6443/v1-k3s/connect"
Dec 12 13:56:03 ip-  k3s[1819]: time="2024-12-12T13:56:03Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 172.31.9.184:6443"
Dec 12 13:56:03 ip- 9 k3s[1819]: time="2024-12-12T13:56:03Z" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [1 3] [default:  ]"
Dec 12 13:56:03 ip-  k3s[1819]: time="2024-12-12T13:56:03Z" level=info msg="Started tunnel to  :6443"


Dec 12 13:56:03 ip-  k3s[1819]: time="2024-12-12T13:56:03Z" level=info msg="Remotedialer connected to proxy" url="wss://172.31.9.184:6443/v1-k3s/connect"
Dec 12 13:56:03 ip-  k3s[1819]: time="2024-12-12T13:56:03Z" level=info msg="Server  @UNCHECKED->RECOVERING from successful health check"
Dec 12 13:56:04 ip-  k3s[1819]: time="2024-12-12T13:56:04Z" level=info msg="Server  @RECOVERING->PREFERRED from successful health check"
Dec 12 13:56:06 ip  k3s[1819]: I1212 13:56:06.045238    1819 kube.go:490] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.42.7.0/24]

• after all steps - ran also upgrade node replacement suite - ok
  https://github.com/rancher/distros-test-framework/blob/main/entrypoint/upgradecluster/upgradenodereplacement_test.go