Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-1.30] loadbalancer improvements #11425

Closed
brandond opened this issue Dec 6, 2024 · 2 comments
Closed

[release-1.30] loadbalancer improvements #11425

brandond opened this issue Dec 6, 2024 · 2 comments
Assignees
@brandond @VestigeJ
Milestone
v1.30.8+k3s1

Comments

@brandond
Copy link
Member

brandond commented Dec 6, 2024

Backport to release-1.30:
@brandond brandond moved this from New to Working in K3s Development Dec 6, 2024
@brandond brandond added this to the v1.30.8+k3s1 milestone Dec 6, 2024
@VestigeJ VestigeJ self-assigned this Dec 9, 2024
@brandond brandond mentioned this issue Dec 10, 2024
Merged
@ShylajaDevadiga ShylajaDevadiga moved this from Working to To Test in K3s Development Dec 11, 2024
@brandond brandond self-assigned this Dec 11, 2024
@ShylajaDevadiga
Copy link
Contributor

#11334 (comment)

@VestigeJ
Copy link

##Environment Details
Validated using COMMIT=fe0a1baef5b084566125cd8c3c80c2bbf3131a53

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME               STATUS   ROLES                  AGE   VERSION
ip-196             Ready    etcd                   55m   v1.30.7+k3s-fe0a1bae
ip-234             Ready    <none>                 55m   v1.30.7+k3s-fe0a1bae
ip-236             Ready    control-plane,master   55m   v1.30.7+k3s-fe0a1bae
ip-25              Ready    control-plane,master   13m   v1.30.7+k3s-fe0a1bae
ip-157             Ready    etcd                   55m   v1.30.7+k3s-fe0a1bae
ip-196             Ready    etcd                   55m   v1.30.7+k3s-fe0a1bae

Config.yaml:

node-external-ip: node.236
server: https://node.196:6443
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
embedded-registry: true
node-ip: node.236
disable-etcd: true

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ COMMIT=fe0a1baef5b084566125cd8c3c80c2bbf3131a53
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh
$ sudo systemctl restart k3s
$ kgn
$ sudo reboot
$ set_kubefig
$ kgn
$ curl -H 'Host: test1.com' http://node.234/name.html
$ kg ing -A
$ curl -H 'Host: test1.com' http://node.236/name.html
$ curl -H 'Host: test1.com' http://node.29/name.html
$ kg ing -A
$ curl -H 'Host: test1.com' http://node.29/name.html
$ curl -H 'Host: test1.com' http://node.234/name.html
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
$ sudo systemctl stop k3s
$ sudo systemctl restart k3s
$ kgn

Results:

$ kg ing -A

DEBU[0000] Asset dir /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152
DEBU[0000] Running /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152/bin/kubectl [kubectl get ing -A]
NAMESPACE   NAME                CLASS     HOSTS       ADDRESS                                                 PORTS   AGE
default     othertest-ingress   traefik   test1.com   node.234,node.236,node.29,node.25   80      41m

$ k delete node ip-node-29

DEBU[0000] Asset dir /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152
DEBU[0000] Running /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152/bin/kubectl [kubectl delete node ip-node-29]
node "ip-node-29" deleted

$ kg ing -A

DEBU[0000] Asset dir /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152
DEBU[0000] Running /var/lib/rancher/k3s/data/d631ec053fa44dd6bfaf32107fcd141e782ee4b1c635405568d7ad5e0dc94152/bin/kubectl [kubectl get ing -A]
NAMESPACE   NAME                CLASS     HOSTS       ADDRESS                                    PORTS   AGE
default     othertest-ingress   traefik   test1.com   node.234,node.236,node.25   80      43m

$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-697kg
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-ml66c
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-gvwzf
$ curl -H 'Host: test1.com' http://node.234:31945/name.html
othertest-deploy-697kg


//after shutting down all control-plane nodes logs from the agent node
$ sudo journalctl -u k3s-agent -f

k3s[1533]: time="2024-12-12T00:35:22Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:22Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:22Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:22Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:23Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:23Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:23Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:23Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:23Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:23Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:24Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:24Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:24Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:24Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:24Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:24Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:25Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:25Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:25Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:25Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:25Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:25Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:26Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:26Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:26Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:26Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:26Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:26Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:27Z" level=warning msg="Failed to get apiserver addresses from supervisor: Get "https://127.0.0.1:6444/v1-k3s/apiservers\": read tcp 127.0.0.1:36996->127.0.0.1:6444: read: connection reset by peer"
k3s[1533]: time="2024-12-12T00:35:27Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:27Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:27Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:27Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:27Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:27Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:28Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:28Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:28Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:28Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:28Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:28Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:29Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:29Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:29Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:29Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:29Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:29Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:30Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:30Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:30Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:30Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:30Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:30Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:31Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:31Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:31Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:31Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:31Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:31Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:32.065084 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="Get "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-172-31-24-234?timeout=10s\": context deadline exceeded - error from a previous attempt: read tcp 127.0.0.1:49282->127.0.0.1:6444: read: connection reset by peer" interval="6.4s"
k3s[1533]: W1212 00:35:32.545907 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Pod: starting
k3s[1533]: E1212 00:35:32.545950 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: starting
k3s[1533]: W1212 00:35:32.594816 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: starting
k3s[1533]: E1212 00:35:32.594953 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: starting
k3s[1533]: W1212 00:35:32.684864 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: starting
k3s[1533]: E1212 00:35:32.684906 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: starting
k3s[1533]: W1212 00:35:32.686017 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Service: starting
k3s[1533]: E1212 00:35:32.686149 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Service: failed to list *v1.Service: starting
k3s[1533]: time="2024-12-12T00:35:32Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:32Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:32Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:32.726226 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Service: starting
k3s[1533]: E1212 00:35:32.726266 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Service: failed to list *v1.Service: starting
k3s[1533]: W1212 00:35:32.799104 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.RuntimeClass: starting
k3s[1533]: E1212 00:35:32.799147 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: starting
k3s[1533]: W1212 00:35:32.835944 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: starting
k3s[1533]: E1212 00:35:32.836048 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: starting
k3s[1533]: W1212 00:35:32.889974 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: starting
k3s[1533]: E1212 00:35:32.890012 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: starting
k3s[1533]: W1212 00:35:32.942459 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.CSIDriver: starting
k3s[1533]: E1212 00:35:32.942497 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: starting
k3s[1533]: time="2024-12-12T00:35:32Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:32Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:32Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:32.993212 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Pod: starting
k3s[1533]: E1212 00:35:32.993252 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: starting
k3s[1533]: W1212 00:35:33.075773 1533 reflector.go:547] object-"default"/"kube-root-ca.crt": failed to list *v1.ConfigMap: apiserver disabled
k3s[1533]: E1212 00:35:33.075811 1533 reflector.go:150] object-"default"/"kube-root-ca.crt": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver disabled
k3s[1533]: W1212 00:35:33.346707 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Namespace: apiserver disabled
k3s[1533]: E1212 00:35:33.346746 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: apiserver disabled
k3s[1533]: W1212 00:35:33.347537 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:35:33.347566 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:35:33.432329 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: W1212 00:35:33.432567 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Endpoints: apiserver disabled
k3s[1533]: E1212 00:35:33.432668 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: apiserver disabled
k3s[1533]: E1212 00:35:33.451590 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:33.472927 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:33.491976 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:33.498196 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:33.498219 1533 kubelet_node_status.go:531] "Unable to update node status" err="update node status exceeds retry count"
k3s[1533]: time="2024-12-12T00:35:33Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:33Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:33Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:33Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:33Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:33Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:34Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:34Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:34Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:34Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:34Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:34Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:35Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:35Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:35Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:35Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:35Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:35Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:36Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:36Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:36Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:36Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:36Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:36Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:37Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:37Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:37Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:37Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:37Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:37Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:38.470712 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="apiserver disabled" interval="7s"
k3s[1533]: time="2024-12-12T00:35:38Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:38Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:38Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:38Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:38Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:38Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:39Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:39Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:39Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:39Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:39Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:39Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:40Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:40Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:40Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:40Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:40Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:40Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:41Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:41Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:41Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:41Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:41Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:41Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:42Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:42Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:42Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:42.862490 1533 webhook.go:275] Failed to make webhook authorizer request: apiserver disabled
k3s[1533]: E1212 00:35:42.862549 1533 server.go:319] "Authorization error" err="apiserver disabled" user="system:serviceaccount:kube-system:metrics-server" verb="get" resource="nodes" subresource="metrics"
k3s[1533]: time="2024-12-12T00:35:42Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:42Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:42Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:43.636574 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:43.642167 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:43.650248 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:43.658873 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:43.666590 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:43.666617 1533 kubelet_node_status.go:531] "Unable to update node status" err="update node status exceeds retry count"
k3s[1533]: time="2024-12-12T00:35:43Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:43Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:43Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:43Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:43Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:43Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:44Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:44Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:44Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:44Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:44Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:44Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:45.481310 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="apiserver disabled" interval="7s"
k3s[1533]: time="2024-12-12T00:35:45Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:45Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:45Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:45Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:45Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:45Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:46.332788 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.EndpointSlice: apiserver disabled
k3s[1533]: E1212 00:35:46.332861 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:46Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:46Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:46Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:46Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:46Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:46Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:47.376772 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.RuntimeClass: apiserver disabled
k3s[1533]: E1212 00:35:47.376810 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: apiserver disabled
k3s[1533]: W1212 00:35:47.687540 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:35:47.687587 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:47Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:47Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:47Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:47Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:47Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:47Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:48Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:48Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:48Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:48Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:48Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:48Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:49Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:49Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:49Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:49.760038 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: apiserver disabled
k3s[1533]: E1212 00:35:49.760091 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:49Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:49Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:49Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:49.994619 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Pod: apiserver disabled
k3s[1533]: E1212 00:35:49.994657 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:50Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:50Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:50Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:50Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:50Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:50Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:51.216997 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.CSIDriver: apiserver disabled
k3s[1533]: E1212 00:35:51.217035 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: apiserver disabled
k3s[1533]: W1212 00:35:51.264735 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Pod: apiserver disabled
k3s[1533]: E1212 00:35:51.264772 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: apiserver disabled
k3s[1533]: W1212 00:35:51.330522 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:35:51.330576 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: apiserver disabled
k3s[1533]: W1212 00:35:51.494295 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Endpoints: apiserver disabled
k3s[1533]: E1212 00:35:51.494566 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:51Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:51Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:51Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:51Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:51Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:51Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:52.102570 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Namespace: apiserver disabled
k3s[1533]: E1212 00:35:52.102612 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: apiserver disabled
k3s[1533]: E1212 00:35:52.486479 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="apiserver disabled" interval="7s"
k3s[1533]: time="2024-12-12T00:35:52Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:52Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:52Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:52.820680 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:35:52.820719 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:52Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:52Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:52Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:53Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:53Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:53Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:53.981985 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: time="2024-12-12T00:35:53Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:53Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:53Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:53.991612 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:54.004017 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:54.010920 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:54.017019 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:35:54.017041 1533 kubelet_node_status.go:531] "Unable to update node status" err="update node status exceeds retry count"
k3s[1533]: W1212 00:35:54.472496 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Service: apiserver disabled
k3s[1533]: E1212 00:35:54.472532 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Service: failed to list *v1.Service: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:54Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:54Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:54Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:54Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:54Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:54Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:55.669627 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Service: apiserver disabled
k3s[1533]: E1212 00:35:55.669668 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Service: failed to list *v1.Service: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:55Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:55Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:55Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:55Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:55Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:55Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:56Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:56Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:56Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:56Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:56Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:56Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:57Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:57Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:57Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:57.836456 1533 webhook.go:275] Failed to make webhook authorizer request: apiserver disabled
k3s[1533]: E1212 00:35:57.836490 1533 server.go:319] "Authorization error" err="apiserver disabled" user="system:serviceaccount:kube-system:metrics-server" verb="get" resource="nodes" subresource="metrics"
k3s[1533]: time="2024-12-12T00:35:57Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:57Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:57Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:35:58.656775 1533 reflector.go:547] object-"default"/"kube-root-ca.crt": failed to list *v1.ConfigMap: apiserver disabled
k3s[1533]: E1212 00:35:58.656814 1533 reflector.go:150] object-"default"/"kube-root-ca.crt": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver disabled
k3s[1533]: time="2024-12-12T00:35:58Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:58Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:58Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:58Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:58Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:58Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:35:59.490367 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="apiserver disabled" interval="7s"
k3s[1533]: time="2024-12-12T00:35:59Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:59Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:59Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:59Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:35:59Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:35:59Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:00Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:00Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:00Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:00Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:00Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:00Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:01Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:01Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:01Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:01Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:01Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:01Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:02Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:02Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:02Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:03Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:36:03Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:36:03Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:38:58Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:38:58.517256 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:38:58.521784 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:38:58.532513 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:38:58.540131 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:38:58.546434 1533 kubelet_node_status.go:544] "Error updating node status, will retry" err="error getting node "ip-172-31-24-234": apiserver disabled"
k3s[1533]: E1212 00:38:58.546457 1533 kubelet_node_status.go:531] "Unable to update node status" err="update node status exceeds retry count"
k3s[1533]: time="2024-12-12T00:38:58Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:38:58Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:38:58Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:38:59Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:38:59Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:38:59Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:38:59Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:38:59Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:38:59Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:00Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:00Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:00Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:00Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:00Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:00Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:01Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:01Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:01Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: E1212 00:39:01.676969 1533 controller.go:145] "Failed to ensure lease exists, will retry" err="apiserver disabled" interval="7s"
k3s[1533]: time="2024-12-12T00:39:01Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:01Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:01Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:02Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:02Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:02Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:02Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:02Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:02Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: W1212 00:39:03.155343 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: apiserver disabled
k3s[1533]: E1212 00:39:03.155389 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: apiserver disabled
k3s[1533]: time="2024-12-12T00:39:03Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:03Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.25:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:03Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.25:6443: connect: connection refused" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: W1212 00:39:03.661846 1533 reflector.go:547] k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.Node: apiserver disabled
k3s[1533]: E1212 00:39:03.661886 1533 reflector.go:150] k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.Node: failed to list *v1.Node: apiserver disabled
k3s[1533]: time="2024-12-12T00:39:03Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:03Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:03Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:04Z" level=info msg="Connecting to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:04Z" level=info msg="Remotedialer connected to proxy" url="wss://node.25:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:04Z" level=info msg="Server node.25:6443@FAILED->RECOVERING from successful health check"
k3s[1533]: time="2024-12-12T00:39:04Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:04Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:04Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:05Z" level=info msg="Server node.25:6443@RECOVERING->PREFERRED from successful health check"
k3s[1533]: time="2024-12-12T00:39:05Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:05Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:05Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:06Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:06Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp node.236:6443: connect: connection refused"
k3s[1533]: time="2024-12-12T00:39:06Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp node.236:6443: connect: connection refused" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:07Z" level=info msg="Connecting to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:07Z" level=info msg="Remotedialer connected to proxy" url="wss://node.236:6443/v1-k3s/connect"
k3s[1533]: time="2024-12-12T00:39:08Z" level=info msg="Server node.25:6443@PREFERRED->ACTIVE from successful dial"
k3s[1533]: time="2024-12-12T00:39:08Z" level=info msg="Server node.236:6443@FAILED->RECOVERING from successful health check"
k3s[1533]: time="2024-12-12T00:39:09Z" level=info msg="Server node.236:6443@RECOVERING->PREFERRED from successful health check"


</details>

@github-project-automation github-project-automation bot moved this from To Test to Done Issue in K3s Development Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
v1.30.8+k3s1
Development

No branches or pull requests
3 participants
@brandond @VestigeJ @ShylajaDevadiga