RKE2 cloud-controller-manager cant reach kube api server, tcp 127.0.0.1:6443: connect: connection refused. Can you help me?

[cengizhanR]

rke2 Version 1.30.4

I0921 07:48:04.499086       1 serving.go:387] Generated self-signed cert in-memory
I0921 07:48:04.685141       1 serving.go:387] Generated self-signed cert in-memory
I0921 07:48:05.058202       1 controllermanager.go:169] Version: v1.29.3-k3s1
I0921 07:48:05.064505       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0921 07:48:05.064654       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0921 07:48:05.064556       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0921 07:48:05.064766       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0921 07:48:05.064570       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0921 07:48:05.064886       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0921 07:48:05.065931       1 secure_serving.go:213] Serving securely on 127.0.0.1:10258
I0921 07:48:05.065980       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0921 07:48:05.067792       1 leaderelection.go:250] attempting to acquire leader lease kube-system/rke2-cloud-controller-manager...
I0921 07:48:05.165004       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0921 07:48:05.165178       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0921 07:48:05.165225       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0921 10:23:45.014675       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:49.194371       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:51.572610       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:55.031845       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:58.502866       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:01.950612       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:04.238134       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:07.938182       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:11.111673       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:13.758526       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:17.737175       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:21.622091       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:24.751748       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:29.220193       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: leases.coordination.k8s.io "rke2-cloud-controller-manager" is forbidden: User "rke2-cloud-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E0921 10:24:46.002224       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: leases.coordination.k8s.io "rke2-cloud-controller-manager" is forbidden: User "rke2-cloud-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E0921 10:25:05.952956       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: leases.coordination.k8s.io "rke2-cloud-controller-manager" is forbidden: User "rke2-cloud-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E0921 10:25:26.002996       1 leaderelection.go:332] error retrieving resource lock kube-system/rke2-cloud-controller-manager: leases.coordination.k8s.io "rke2-cloud-controller-manager" is forbidden: User "rke2-cloud-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

[cengizhanR]

This is kube-controller-manager logs there are problems too.

`I0921 07:47:35.478155       1 serving.go:380] Generated self-signed cert in-memory
I0921 07:47:35.677113       1 controllermanager.go:189] "Starting" version="v1.30.4+rke2r1"
I0921 07:47:35.677135       1 controllermanager.go:191] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0921 07:47:35.685556       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0921 07:47:35.685572       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0921 07:47:35.685587       1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I0921 07:47:35.686040       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0921 07:47:35.686105       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0921 07:47:35.686119       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0921 07:47:35.685604       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0921 07:47:35.685801       1 secure_serving.go:213] Serving securely on 127.0.0.1:10257
I0921 07:47:35.687461       1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-controller-manager...
I0921 07:47:35.786689       1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I0921 07:47:35.786737       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0921 07:47:35.787877       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E0921 10:23:46.337056       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:48.822991       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:51.919414       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:55.877666       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:23:58.307000       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:01.971886       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:04.170074       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:06.311305       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:09.589347       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:12.406496       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:15.972181       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:18.621411       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:22.258828       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:24.478324       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:24:28.005580       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:25:04.504867       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s": dial tcp 127.0.0.1:6443: connect: connection refused
W0921 10:25:24.729126       1 reflector.go:547] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: failed to list *v1.ConfigMap: Get "https://127.0.0.1:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=5981002": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:25:24.729184       1 reflector.go:150] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://127.0.0.1:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=5981002": dial tcp 127.0.0.1:6443: connect: connection refused
E0921 10:25:26.001153       1 leaderelection.go:347] error retrieving resource lock kube-system/kube-controller-manager: leases.coordination.k8s.io "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"`

[cengizhanR]

This is config yaml for rke2-server

[root@SBLG-PISRV01 bilgedeployment]# cat /etc/rancher/rke2/config.yaml
secrets-encryption: true
token: fuzzybunnyslippers
tls-san:
  - 
  - 172.16.16.64
node-ip: "10.224.83.61"
etcd-expose-metrics: true
write-kubeconfig-mode: "0600"
use-service-account-credentials: true
node-taint:
- "CriticalAddonsOnly=true:NoExecute"
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-proxy-arg:
- metrics-bind-address=0.0.0.0:10249
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- audit-policy-file=/etc/rancher/rke2/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
- audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
- "container-log-max-files=5"
- "container-log-max-size=10Mi"

[brandond]

It looks like the apiserver was just down for a bit. The connection error goes away, then you got RBAC errors, and then the errors stop completely and it appears to be working fine. Was this during the initial startup of this node or cluster, and it was just waiting for etcd and the apiserver to come up and RBAC to be established?

[cengizhanR]

@brandond Thank you for your return.
I just realized that this may be the case when it is first installed. I have two clusters and this is the case in both of them and I have not restarted any of the nodes.
I sent requests to the Kube API servers manually, everything seems to be working properly. But these logs always remain like this. I restarted all servers with the "systemctl restart rke2-server" command, but the logs did not change.

[brandond]

The logs aren't changing because the pods are still running and working fine. Pods aren't restarted just because you restart the service, they remain running across restarts. If you want to clear the logs you could always delete the pods so that they will be recreated, but there's not really any point in doing that other than just satisfying your own desire to see the old errors go away.

[cengizhanR]

@brandond Thank you very much for your interest and help.