Yesterday, one of my master nodes crashed. I am sharing the Kuebelet logs. Can you help? RKE2-server status failed. fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" #6529

cengizhanR · 2024-08-10T07:44:24Z

cengizhanR
Aug 10, 2024

Iam using Kubernetes Version: v1.30.0 +rke2r1

failed to destroy network for sandbox \"437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: Get \"https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: connect: connection timed out" podSandboxID="437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463"

fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s\": proxy error from 127.0.0.1:9345 while dialing 10.42.4.12:9443, code 502: 502 Bad Gateway"


[root@rke21 logs]# cat kubelet.log | tail -n 300
E0809 15:02:12.403492 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.504190 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.604951 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.705936 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.806347 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.907403 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.008131 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.108719 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.209203 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.309627 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.409752 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.510334 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.611036 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.711941 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.812066 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.912744 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.013544 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.114439 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:09.350873 1845966 event.go:389] "Event occurred" object="kube-system/rke2-metrics-server" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
I0809 15:02:09.367409 1845966 event.go:389] "Event occurred" object="kube-system/rke2-metrics-server" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
I0809 15:02:09.548180 1845966 event.go:389] "Event occurred" object="kube-system/rke2-multus" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
I0809 15:02:09.769593 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller-crd" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml\""
I0809 15:02:09.782142 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller-crd" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml\""
I0809 15:02:09.953210 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml\""
I0809 15:02:09.966150 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml\""
I0809 15:02:10.147026 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-validation-webhook" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml\""
I0809 15:02:10.160054 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-validation-webhook" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml\""
E0809 15:02:14.214920 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.315237 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.347481 1846549 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:14.415545 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.516173 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.616961 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.717107 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.818033 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.918812 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.019680 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.120479 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.221302 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.322200 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.423364 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.523812 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.624576 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.725647 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.826272 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.926601 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.027169 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.127871 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.228840 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.329172 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.429633 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.530204 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.630344 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.730828 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.831684 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.932249 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.032911 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.133838 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.234481 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.335020 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.435667 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.536742 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.637178 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.737759 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.838806 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.939821 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.040536 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.141351 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.241789 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.342978 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.444166 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.545040 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.645651 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.746642 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.846867 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.947831 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.048870 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.149870 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:16.990861 1845966 event.go:389] "Event occurred" object="rke21" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s\": proxy error from 127.0.0.1:9345 while dialing 10.42.4.12:9443, code 502: 502 Bad Gateway"
E0809 15:02:19.250487 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.351037 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.451957 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.552883 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.652981 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.753628 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.853982 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.954823 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.055303 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.156120 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.256856 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.357457 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.458073 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.558913 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.659640 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.760802 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.861940 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.927918 1846549 resource_metrics.go:161] "Error getting summary for resourceMetric prometheus endpoint" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:20.962078 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.063067 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.163251 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.264403 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.365122 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.465281 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.565959 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.666819 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.767698 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.868223 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.969128 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.069877 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.170719 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.271475 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.371961 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.472802 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.572936 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.673859 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.774382 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.875427 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.976053 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.076906 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.177464 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.278325 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.379040 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.479991 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.580799 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.681508 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.782044 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.882419 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.982957 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.083548 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.183658 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:19.918742 1845966 event.go:389] "Event occurred" object="rke21" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidationComplete" message="Deferred node password secret validation complete"
E0809 15:02:24.283825 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.348401 1846549 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:24.384744 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.485377 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.586279 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.686781 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.787214 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.887801 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.988580 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.089410 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.190500 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.291306 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.391696 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.492410 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.592663 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.693185 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.794325 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.894835 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.995389 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.096019 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.196950 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.297299 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.397424 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.497595 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.598326 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.698963 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.799395 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.899883 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.000887 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.101474 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.202042 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.302174 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.402562 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.503054 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.603333 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.704388 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.804576 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.905425 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.006340 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.106866 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.207640 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.307987 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.408437 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.509458 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.610217 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.710968 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.811793 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.912487 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.013116 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.114109 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.214739 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.315112 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.416148 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.516762 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.617636 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.717835 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.818555 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.919550 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:30.020112 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:30.120423 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:30.323615 1846549 apiserver.go:52] "Watching apiserver"
I0809 15:02:30.335123 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="06a98001-8765-4306-8ff2-e5c5d104e360" podNamespace="kube-system" podName="rke2-canal-67pb9"
I0809 15:02:30.335390 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="4d78ab62-e2c1-4a0e-bc14-0f8d4d964548" podNamespace="cattle-monitoring-system" podName="pushprox-kube-controller-manager-client-h2ppn"
I0809 15:02:30.335493 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="f9258daf-96fd-4d3f-b2ed-a458df4228bb" podNamespace="cattle-monitoring-system" podName="rancher-monitoring-prometheus-node-exporter-l2fnl"
I0809 15:02:30.335605 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="c3172ee9-5576-4463-b059-10590f1e744b" podNamespace="cattle-monitoring-system" podName="pushprox-kube-etcd-client-zk4jm"
I0809 15:02:30.336027 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="eb800811-7140-4738-be1c-543022924686" podNamespace="cattle-monitoring-system" podName="pushprox-kube-proxy-client-r2945"
I0809 15:02:30.336150 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="1aa085cf-1222-4996-867c-34440cc1bc33" podNamespace="cattle-monitoring-system" podName="pushprox-kube-scheduler-client-kk24v"
I0809 15:02:30.336243 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="aa7414ab-15c7-4e78-8ff7-0ad16112445d" podNamespace="kube-system" podName="rke2-ingress-nginx-controller-mfhqd"
I0809 15:02:30.336372 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="901de050-3c5b-4ada-a4d4-38c607ae65de" podNamespace="cattle-provisioning-capi-system" podName="capi-controller-manager-745f484f4d-2pjnp"
I0809 15:02:30.336458 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="ddd2c993-97db-49be-8b89-5643e77075cc" podNamespace="cattle-monitoring-system" podName="pushprox-kube-etcd-proxy-6bf9dfb758-bntmr"
I0809 15:02:30.336783 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="4445ef1d-0567-416f-aa69-81c36ea28a38" podNamespace="cert-manager" podName="cert-manager-cainjector-6ff697db96-bvq9h"
I0809 15:02:30.336890 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98" podNamespace="cengiz" podName="nginx-deployment-8564b58ddc-xbqx8"
I0809 15:02:30.337082 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a62a6cfa-93c7-48bf-807f-a485c8f52549" podNamespace="cattle-monitoring-system" podName="pushprox-kube-scheduler-proxy-8d54cf848-v49wv"
I0809 15:02:30.337182 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a76f5364-62f3-46d8-9e01-f40863e6daf0" podNamespace="cattle-monitoring-system" podName="pushprox-kube-controller-manager-proxy-7cdd86d77d-vt5fm"
I0809 15:02:30.337355 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8" podNamespace="cattle-logging-system" podName="rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.390624 1846549 desired_state_of_world_populator.go:157] "Finished populating initial desired state of world"
I0809 15:02:30.479627 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t926k\" (UniqueName: \"kubernetes.io/projected/1aa085cf-1222-4996-867c-34440cc1bc33-kube-api-access-t926k\") pod \"pushprox-kube-scheduler-client-kk24v\" (UID: \"1aa085cf-1222-4996-867c-34440cc1bc33\") " pod="cattle-monitoring-system/pushprox-kube-scheduler-client-kk24v"
I0809 15:02:30.479676 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-calico\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-var-run-calico\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.479710 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-log-dir\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-cni-log-dir\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.479749 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjkb\" (UniqueName: \"kubernetes.io/projected/ddd2c993-97db-49be-8b89-5643e77075cc-kube-api-access-shjkb\") pod \"pushprox-kube-etcd-proxy-6bf9dfb758-bntmr\" (UID: \"ddd2c993-97db-49be-8b89-5643e77075cc\") " pod="cattle-monitoring-system/pushprox-kube-etcd-proxy-6bf9dfb758-bntmr"
I0809 15:02:30.481547 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthw7\" (UniqueName: \"kubernetes.io/projected/4d78ab62-e2c1-4a0e-bc14-0f8d4d964548-kube-api-access-sthw7\") pod \"pushprox-kube-controller-manager-client-h2ppn\" (UID: \"4d78ab62-e2c1-4a0e-bc14-0f8d4d964548\") " pod="cattle-monitoring-system/pushprox-kube-controller-manager-client-h2ppn"
I0809 15:02:30.481612 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-calico\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-var-lib-calico\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.481711 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp72x\" (UniqueName: \"kubernetes.io/projected/06a98001-8765-4306-8ff2-e5c5d104e360-kube-api-access-mp72x\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.481747 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68fd\" (UniqueName: \"kubernetes.io/projected/a62a6cfa-93c7-48bf-807f-a485c8f52549-kube-api-access-g68fd\") pod \"pushprox-kube-scheduler-proxy-8d54cf848-v49wv\" (UID: \"a62a6cfa-93c7-48bf-807f-a485c8f52549\") " pod="cattle-monitoring-system/pushprox-kube-scheduler-proxy-8d54cf848-v49wv"
I0809 15:02:30.481779 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm4j\" (UniqueName: \"kubernetes.io/projected/a76f5364-62f3-46d8-9e01-f40863e6daf0-kube-api-access-hqm4j\") pod \"pushprox-kube-controller-manager-proxy-7cdd86d77d-vt5fm\" (UID: \"a76f5364-62f3-46d8-9e01-f40863e6daf0\") " pod="cattle-monitoring-system/pushprox-kube-controller-manager-proxy-7cdd86d77d-vt5fm"
I0809 15:02:30.481809 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa7414ab-15c7-4e78-8ff7-0ad16112445d-webhook-cert\") pod \"rke2-ingress-nginx-controller-mfhqd\" (UID: \"aa7414ab-15c7-4e78-8ff7-0ad16112445d\") " pod="kube-system/rke2-ingress-nginx-controller-mfhqd"
I0809 15:02:30.481842 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9f4\" (UniqueName: \"kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4\") pod \"nginx-deployment-8564b58ddc-xbqx8\" (UID: \"a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98\") " pod="cengiz/nginx-deployment-8564b58ddc-xbqx8"
I0809 15:02:30.481872 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9258daf-96fd-4d3f-b2ed-a458df4228bb-root\") pod \"rancher-monitoring-prometheus-node-exporter-l2fnl\" (UID: \"f9258daf-96fd-4d3f-b2ed-a458df4228bb\") " pod="cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-l2fnl"
I0809 15:02:30.481903 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfkb\" (UniqueName: \"kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb\") pod \"cert-manager-cainjector-6ff697db96-bvq9h\" (UID: \"4445ef1d-0567-416f-aa69-81c36ea28a38\") " pod="cert-manager/cert-manager-cainjector-6ff697db96-bvq9h"
I0809 15:02:30.481937 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-xtables-lock\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.481974 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8-config\") pod \"rancher-logging-root-fluentbit-t2fbq\" (UID: \"e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8\") " pod="cattle-logging-system/rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.482003 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlibcontainers\" (UniqueName: \"kubernetes.io/host-path/e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8-varlibcontainers\") pod \"rancher-logging-root-fluentbit-t2fbq\" (UID: \"e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8\") " pod="cattle-logging-system/rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.482030 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"vol-m33oo\" (UniqueName: \"kubernetes.io/nfs/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-vol-m33oo\") pod \"nginx-deployment-8564b58ddc-xbqx8\" (UID: \"a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98\") " pod="cengiz/nginx-deployment-8564b58ddc-xbqx8"
I0809 15:02:30.482064 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogs\" (UniqueName: \"kubernetes.io/host-path/e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8-varlogs\") pod \"rancher-logging-root-fluentbit-t2fbq\" (UID: \"e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8\") " pod="cattle-logging-system/rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.482099 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9258daf-96fd-4d3f-b2ed-a458df4228bb-proc\") pod \"rancher-monitoring-prometheus-node-exporter-l2fnl\" (UID: \"f9258daf-96fd-4d3f-b2ed-a458df4228bb\") " pod="cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-l2fnl"
I0809 15:02:30.482192 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-lib-modules\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.482266 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-bin-dir\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-cni-bin-dir\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.482376 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm5w\" (UniqueName: \"kubernetes.io/projected/aa7414ab-15c7-4e78-8ff7-0ad16112445d-kube-api-access-xgm5w\") pod \"rke2-ingress-nginx-controller-mfhqd\" (UID: \"aa7414ab-15c7-4e78-8ff7-0ad16112445d\") " pod="kube-system/rke2-ingress-nginx-controller-mfhqd"
I0809 15:02:30.483024 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncpv\" (UniqueName: \"kubernetes.io/projected/eb800811-7140-4738-be1c-543022924686-kube-api-access-gncpv\") pod \"pushprox-kube-proxy-client-r2945\" (UID: \"eb800811-7140-4738-be1c-543022924686\") " pod="cattle-monitoring-system/pushprox-kube-proxy-client-r2945"
I0809 15:02:30.483084 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx9l\" (UniqueName: \"kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l\") pod \"capi-controller-manager-745f484f4d-2pjnp\" (UID: \"901de050-3c5b-4ada-a4d4-38c607ae65de\") " pod="cattle-provisioning-capi-system/capi-controller-manager-745f484f4d-2pjnp"
I0809 15:02:30.483129 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-net-dir\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-cni-net-dir\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.483157 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"policysync\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-policysync\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.483214 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9258daf-96fd-4d3f-b2ed-a458df4228bb-sys\") pod \"rancher-monitoring-prometheus-node-exporter-l2fnl\" (UID: \"f9258daf-96fd-4d3f-b2ed-a458df4228bb\") " pod="cattle-monitoring-system/rancher-monitoring-prometheus-node-exporter-l2fnl"
I0809 15:02:30.483359 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flexvol-driver-host\" (UniqueName: \"kubernetes.io/host-path/06a98001-8765-4306-8ff2-e5c5d104e360-flexvol-driver-host\") pod \"rke2-canal-67pb9\" (UID: \"06a98001-8765-4306-8ff2-e5c5d104e360\") " pod="kube-system/rke2-canal-67pb9"
I0809 15:02:30.483566 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kqb\" (UniqueName: \"kubernetes.io/projected/c3172ee9-5576-4463-b059-10590f1e744b-kube-api-access-64kqb\") pod \"pushprox-kube-etcd-client-zk4jm\" (UID: \"c3172ee9-5576-4463-b059-10590f1e744b\") " pod="cattle-monitoring-system/pushprox-kube-etcd-client-zk4jm"
I0809 15:02:30.483722 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862q2\" (UniqueName: \"kubernetes.io/projected/e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8-kube-api-access-862q2\") pod \"rancher-logging-root-fluentbit-t2fbq\" (UID: \"e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8\") " pod="cattle-logging-system/rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.483759 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert\") pod \"capi-controller-manager-745f484f4d-2pjnp\" (UID: \"901de050-3c5b-4ada-a4d4-38c607ae65de\") " pod="cattle-provisioning-capi-system/capi-controller-manager-745f484f4d-2pjnp"
E0809 15:02:30.596223 1846549 secret.go:194] Couldn't get secret cattle-provisioning-capi-system/capi-webhook-service-cert: object "cattle-provisioning-capi-system"/"capi-webhook-service-cert" not registered
E0809 15:02:30.596335 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert podName:901de050-3c5b-4ada-a4d4-38c607ae65de nodeName:}" failed. No retries permitted until 2024-08-09 15:02:31.096304552 +0300 +03 m=+66.899892406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert") pod "capi-controller-manager-745f484f4d-2pjnp" (UID: "901de050-3c5b-4ada-a4d4-38c607ae65de") : object "cattle-provisioning-capi-system"/"capi-webhook-service-cert" not registered
E0809 15:02:30.623452 1846549 projected.go:294] Couldn't get configMap cattle-provisioning-capi-system/kube-root-ca.crt: object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:30.623496 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-7nx9l for pod cattle-provisioning-capi-system/capi-controller-manager-745f484f4d-2pjnp: object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:30.623579 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l podName:901de050-3c5b-4ada-a4d4-38c607ae65de nodeName:}" failed. No retries permitted until 2024-08-09 15:02:31.123552675 +0300 +03 m=+66.927140629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7nx9l" (UniqueName: "kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l") pod "capi-controller-manager-745f484f4d-2pjnp" (UID: "901de050-3c5b-4ada-a4d4-38c607ae65de") : object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:30.644113 1846549 projected.go:294] Couldn't get configMap cert-manager/kube-root-ca.crt: object "cert-manager"/"kube-root-ca.crt" not registered
E0809 15:02:30.644167 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-6tfkb for pod cert-manager/cert-manager-cainjector-6ff697db96-bvq9h: object "cert-manager"/"kube-root-ca.crt" not registered
E0809 15:02:30.644282 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb podName:4445ef1d-0567-416f-aa69-81c36ea28a38 nodeName:}" failed. No retries permitted until 2024-08-09 15:02:31.144208195 +0300 +03 m=+66.947796149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6tfkb" (UniqueName: "kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb") pod "cert-manager-cainjector-6ff697db96-bvq9h" (UID: "4445ef1d-0567-416f-aa69-81c36ea28a38") : object "cert-manager"/"kube-root-ca.crt" not registered
E0809 15:02:30.645367 1846549 projected.go:294] Couldn't get configMap cengiz/kube-root-ca.crt: object "cengiz"/"kube-root-ca.crt" not registered
E0809 15:02:30.645385 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-4d9f4 for pod cengiz/nginx-deployment-8564b58ddc-xbqx8: object "cengiz"/"kube-root-ca.crt" not registered
E0809 15:02:30.645426 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4 podName:a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98 nodeName:}" failed. No retries permitted until 2024-08-09 15:02:31.145410614 +0300 +03 m=+66.948998568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4d9f4" (UniqueName: "kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4") pod "nginx-deployment-8564b58ddc-xbqx8" (UID: "a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98") : object "cengiz"/"kube-root-ca.crt" not registered
I0809 15:02:30.939978 1846549 scope.go:117] "RemoveContainer" containerID="49d08a0ebb2d2f35cf20efe7c9424c0e85e6f2d075c58b5ba3951fcfd2ec3e4f"
I0809 15:02:30.950865 1846549 scope.go:117] "RemoveContainer" containerID="d870aef4ef17975e9aab9c9f8d648eb14f03ac10161e667ac6bc1db856b47430"
I0809 15:02:31.192224 1846549 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"vol-m33oo\" (UniqueName: \"kubernetes.io/nfs/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-vol-m33oo\") pod \"a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98\" (UID: \"a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98\") "
E0809 15:02:31.192916 1846549 projected.go:294] Couldn't get configMap cengiz/kube-root-ca.crt: object "cengiz"/"kube-root-ca.crt" not registered
E0809 15:02:31.192945 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-4d9f4 for pod cengiz/nginx-deployment-8564b58ddc-xbqx8: object "cengiz"/"kube-root-ca.crt" not registered
E0809 15:02:31.192988 1846549 secret.go:194] Couldn't get secret cattle-provisioning-capi-system/capi-webhook-service-cert: object "cattle-provisioning-capi-system"/"capi-webhook-service-cert" not registered
E0809 15:02:31.192998 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4 podName:a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98 nodeName:}" failed. No retries permitted until 2024-08-09 15:02:32.192978207 +0300 +03 m=+67.996566061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4d9f4" (UniqueName: "kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4") pod "nginx-deployment-8564b58ddc-xbqx8" (UID: "a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98") : object "cengiz"/"kube-root-ca.crt" not registered
E0809 15:02:31.193041 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert podName:901de050-3c5b-4ada-a4d4-38c607ae65de nodeName:}" failed. No retries permitted until 2024-08-09 15:02:32.193025008 +0300 +03 m=+67.996612862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert") pod "capi-controller-manager-745f484f4d-2pjnp" (UID: "901de050-3c5b-4ada-a4d4-38c607ae65de") : object "cattle-provisioning-capi-system"/"capi-webhook-service-cert" not registered
E0809 15:02:31.192920 1846549 projected.go:294] Couldn't get configMap cattle-provisioning-capi-system/kube-root-ca.crt: object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:31.193061 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-7nx9l for pod cattle-provisioning-capi-system/capi-controller-manager-745f484f4d-2pjnp: object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:31.193091 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l podName:901de050-3c5b-4ada-a4d4-38c607ae65de nodeName:}" failed. No retries permitted until 2024-08-09 15:02:32.193081008 +0300 +03 m=+67.996668962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nx9l" (UniqueName: "kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l") pod "capi-controller-manager-745f484f4d-2pjnp" (UID: "901de050-3c5b-4ada-a4d4-38c607ae65de") : object "cattle-provisioning-capi-system"/"kube-root-ca.crt" not registered
E0809 15:02:31.193220 1846549 projected.go:294] Couldn't get configMap cert-manager/kube-root-ca.crt: object "cert-manager"/"kube-root-ca.crt" not registered
E0809 15:02:31.193232 1846549 projected.go:200] Error preparing data for projected volume kube-api-access-6tfkb for pod cert-manager/cert-manager-cainjector-6ff697db96-bvq9h: object "cert-manager"/"kube-root-ca.crt" not registered
E0809 15:02:31.193401 1846549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb podName:4445ef1d-0567-416f-aa69-81c36ea28a38 nodeName:}" failed. No retries permitted until 2024-08-09 15:02:32.193388713 +0300 +03 m=+67.996976567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tfkb" (UniqueName: "kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb") pod "cert-manager-cainjector-6ff697db96-bvq9h" (UID: "4445ef1d-0567-416f-aa69-81c36ea28a38") : object "cert-manager"/"kube-root-ca.crt" not registered
I0809 15:02:31.229243 1846549 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/nfs/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-vol-m33oo" (OuterVolumeSpecName: "vol-m33oo") pod "a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98" (UID: "a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98"). InnerVolumeSpecName "vol-m33oo". PluginName "kubernetes.io/nfs", VolumeGidValue ""
I0809 15:02:31.294548 1846549 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shjkb\" (UniqueName: \"kubernetes.io/projected/ddd2c993-97db-49be-8b89-5643e77075cc-kube-api-access-shjkb\") pod \"ddd2c993-97db-49be-8b89-5643e77075cc\" (UID: \"ddd2c993-97db-49be-8b89-5643e77075cc\") "
I0809 15:02:31.294686 1846549 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g68fd\" (UniqueName: \"kubernetes.io/projected/a62a6cfa-93c7-48bf-807f-a485c8f52549-kube-api-access-g68fd\") pod \"a62a6cfa-93c7-48bf-807f-a485c8f52549\" (UID: \"a62a6cfa-93c7-48bf-807f-a485c8f52549\") "
I0809 15:02:31.298323 1846549 reconciler_common.go:289] "Volume detached for volume \"vol-m33oo\" (UniqueName: \"kubernetes.io/nfs/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-vol-m33oo\") on node \"rke21\" DevicePath \"\""
I0809 15:02:31.307057 1846549 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62a6cfa-93c7-48bf-807f-a485c8f52549-kube-api-access-g68fd" (OuterVolumeSpecName: "kube-api-access-g68fd") pod "a62a6cfa-93c7-48bf-807f-a485c8f52549" (UID: "a62a6cfa-93c7-48bf-807f-a485c8f52549"). InnerVolumeSpecName "kube-api-access-g68fd". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0809 15:02:31.313966 1846549 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd2c993-97db-49be-8b89-5643e77075cc-kube-api-access-shjkb" (OuterVolumeSpecName: "kube-api-access-shjkb") pod "ddd2c993-97db-49be-8b89-5643e77075cc" (UID: "ddd2c993-97db-49be-8b89-5643e77075cc"). InnerVolumeSpecName "kube-api-access-shjkb". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0809 15:02:31.399461 1846549 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqm4j\" (UniqueName: \"kubernetes.io/projected/a76f5364-62f3-46d8-9e01-f40863e6daf0-kube-api-access-hqm4j\") pod \"a76f5364-62f3-46d8-9e01-f40863e6daf0\" (UID: \"a76f5364-62f3-46d8-9e01-f40863e6daf0\") "
I0809 15:02:31.399655 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-g68fd\" (UniqueName: \"kubernetes.io/projected/a62a6cfa-93c7-48bf-807f-a485c8f52549-kube-api-access-g68fd\") on node \"rke21\" DevicePath \"\""
I0809 15:02:31.399669 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-shjkb\" (UniqueName: \"kubernetes.io/projected/ddd2c993-97db-49be-8b89-5643e77075cc-kube-api-access-shjkb\") on node \"rke21\" DevicePath \"\""
I0809 15:02:31.403340 1846549 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76f5364-62f3-46d8-9e01-f40863e6daf0-kube-api-access-hqm4j" (OuterVolumeSpecName: "kube-api-access-hqm4j") pod "a76f5364-62f3-46d8-9e01-f40863e6daf0" (UID: "a76f5364-62f3-46d8-9e01-f40863e6daf0"). InnerVolumeSpecName "kube-api-access-hqm4j". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0809 15:02:31.500756 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-hqm4j\" (UniqueName: \"kubernetes.io/projected/a76f5364-62f3-46d8-9e01-f40863e6daf0-kube-api-access-hqm4j\") on node \"rke21\" DevicePath \"\""
I0809 15:02:32.012015 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-6tfkb\" (UniqueName: \"kubernetes.io/projected/4445ef1d-0567-416f-aa69-81c36ea28a38-kube-api-access-6tfkb\") on node \"rke21\" DevicePath \"\""
I0809 15:02:32.012050 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-7nx9l\" (UniqueName: \"kubernetes.io/projected/901de050-3c5b-4ada-a4d4-38c607ae65de-kube-api-access-7nx9l\") on node \"rke21\" DevicePath \"\""
I0809 15:02:32.012061 1846549 reconciler_common.go:289] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/901de050-3c5b-4ada-a4d4-38c607ae65de-cert\") on node \"rke21\" DevicePath \"\""
I0809 15:02:32.112825 1846549 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-4d9f4\" (UniqueName: \"kubernetes.io/projected/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98-kube-api-access-4d9f4\") on node \"rke21\" DevicePath \"\""
I0809 15:02:32.318669 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4445ef1d-0567-416f-aa69-81c36ea28a38" path="/var/lib/kubelet/pods/4445ef1d-0567-416f-aa69-81c36ea28a38/volumes"
I0809 15:02:32.319529 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901de050-3c5b-4ada-a4d4-38c607ae65de" path="/var/lib/kubelet/pods/901de050-3c5b-4ada-a4d4-38c607ae65de/volumes"
I0809 15:02:32.320374 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62a6cfa-93c7-48bf-807f-a485c8f52549" path="/var/lib/kubelet/pods/a62a6cfa-93c7-48bf-807f-a485c8f52549/volumes"
I0809 15:02:32.321053 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76f5364-62f3-46d8-9e01-f40863e6daf0" path="/var/lib/kubelet/pods/a76f5364-62f3-46d8-9e01-f40863e6daf0/volumes"
I0809 15:02:32.321631 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98" path="/var/lib/kubelet/pods/a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98/volumes"
I0809 15:02:32.322383 1846549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd2c993-97db-49be-8b89-5643e77075cc" path="/var/lib/kubelet/pods/ddd2c993-97db-49be-8b89-5643e77075cc/volumes"
I0809 15:02:33.758385 1846549 scope.go:117] "RemoveContainer" containerID="f98d2bdb31c160fb0f6e8c1357ec04243a8c5cd87108404953cd5aa89b3c5146"
I0809 15:02:33.758408 1846549 scope.go:117] "RemoveContainer" containerID="1862a4ad94ca42eb05c5935f1bf41da17eaf1c1b887e99820564f7a5a0c622bb"
I0809 15:03:07.610555 1846549 scope.go:117] "RemoveContainer" containerID="a639320241a67572bf6dcf815fee64937875816a6d32b421daf7e9832931ae53"
I0809 15:03:07.618744 1846549 scope.go:117] "RemoveContainer" containerID="91b540b878df9cee9342b8d86e2e738abb5d3dbf1cbf5039be406c250d55a77a"
I0809 15:03:07.624753 1846549 scope.go:117] "RemoveContainer" containerID="de45c71c4787b880c5d54e74f4fb62e2c5d0bc1a45a7adecae69a80f03b526b7"
I0809 15:03:07.630957 1846549 scope.go:117] "RemoveContainer" containerID="6b08e91fc2ed999bca9f6c06db6edb8ecdd2803e3c41e95d3a2754d664bd8a35"
I0809 15:03:07.636679 1846549 scope.go:117] "RemoveContainer" containerID="a76418260032bfe13d2b4883936d906044ee97b18d290a48b1cef3624a929dd3"
I0809 15:03:07.642304 1846549 scope.go:117] "RemoveContainer" containerID="58521296b9c373fc8c7bf23aae532681255f4513f596ae23ac9e8044b733ee9c"
I0809 15:03:07.649228 1846549 scope.go:117] "RemoveContainer" containerID="8f6ed3a4bb8cc3de39c97603d05f652c28fac1cf31a8fbca5f86149a7eaa0a00"
I0809 15:03:07.655810 1846549 scope.go:117] "RemoveContainer" containerID="f1022c29b8b01992843a7d4df8330424a8eb44f1a118225eda3bda00847921ee"
I0809 15:03:07.662325 1846549 scope.go:117] "RemoveContainer" containerID="34d923401a2669b3065f77d29e1d02c9d208320beee2b1ee8cf5ecfd712adcde"
I0809 15:03:07.673752 1846549 scope.go:117] "RemoveContainer" containerID="56701705283195a40647ae8e65b7014332b1030109e22d0c843592a1c36bc5c1"
I0809 15:03:07.683983 1846549 scope.go:117] "RemoveContainer" containerID="0bb975d3db9d69514fcbdc0aa28398d970d2c6057075602e56cbe4349da922d8"
I0809 15:03:07.694669 1846549 scope.go:117] "RemoveContainer" containerID="a5deede12908f993ceb4bfa6c6103437ac285071d10e5c3bce45002d17615848"
E0809 15:12:00.731553 1846549 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="405a9fbbe21def2295e81e09a46d5d1681cf5e5d4e434d5452361f2237f29a1f"
I0810 00:00:05.279708 1845966 event.go:389] "Event occurred" object="local-etcd-snapshot-rke21-1723237205-b6a696" fieldPath="" kind="ETCDSnapshotFile" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ETCDSnapshotCreated" message="Snapshot etcd-snapshot-rke21-1723237205 saved on rke21"
[root@rke21 logs]# cat kubelet.log | tail -n 400
I0809 15:02:05.896912 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file1\" (UniqueName: \"kubernetes.io/host-path/21d1802fa8f81c589e0f3a7d1c617352-file1\") pod \"kube-controller-manager-rke21\" (UID: \"21d1802fa8f81c589e0f3a7d1c617352\") " pod="kube-system/kube-controller-manager-rke21"
I0809 15:02:05.896943 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file2\" (UniqueName: \"kubernetes.io/host-path/8dd8093122d59caedc9f10355507c326-file2\") pod \"cloud-controller-manager-rke21\" (UID: \"8dd8093122d59caedc9f10355507c326\") " pod="kube-system/cloud-controller-manager-rke21"
I0809 15:02:05.896973 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file4\" (UniqueName: \"kubernetes.io/host-path/8dd8093122d59caedc9f10355507c326-file4\") pod \"cloud-controller-manager-rke21\" (UID: \"8dd8093122d59caedc9f10355507c326\") " pod="kube-system/cloud-controller-manager-rke21"
I0809 15:02:05.897002 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file9\" (UniqueName: \"kubernetes.io/host-path/21d1802fa8f81c589e0f3a7d1c617352-file9\") pod \"kube-controller-manager-rke21\" (UID: \"21d1802fa8f81c589e0f3a7d1c617352\") " pod="kube-system/kube-controller-manager-rke21"
I0809 15:02:05.897024 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dir0\" (UniqueName: \"kubernetes.io/host-path/8dd8093122d59caedc9f10355507c326-dir0\") pod \"cloud-controller-manager-rke21\" (UID: \"8dd8093122d59caedc9f10355507c326\") " pod="kube-system/cloud-controller-manager-rke21"
I0809 15:02:05.897057 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file3\" (UniqueName: \"kubernetes.io/host-path/21d1802fa8f81c589e0f3a7d1c617352-file3\") pod \"kube-controller-manager-rke21\" (UID: \"21d1802fa8f81c589e0f3a7d1c617352\") " pod="kube-system/kube-controller-manager-rke21"
E0809 15:02:05.948726 1846549 resource_metrics.go:161] "Error getting summary for resourceMetric prometheus endpoint" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:05.951351 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.051895 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.151994 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.255545 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.356538 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.457411 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.557857 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.658626 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.758774 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.859680 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:06.960412 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.061888 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.162519 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.263537 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.364591 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.465406 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.571328 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.587131 1846549 remote_runtime.go:222] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: Get \"https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: connect: connection timed out" podSandboxID="437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463"
E0809 15:02:07.587173 1846549 kuberuntime_gc.go:180] "Failed to stop sandbox before removing" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: Get \"https://10.43.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: connect: connection timed out" sandboxID="437860a5f84c942859565cea86b92133e8cc40628a6a4d81f5c1b2c1c13e4463"
E0809 15:02:07.672208 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.772841 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.873596 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:07.974799 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.075367 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.175441 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.275972 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.376409 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.477073 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.577567 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.677684 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.778803 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:08.879819 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
Log file created at: 2024/08/09 15:02:08
Running on machine: rke21
Binary: Built with gc go1.22.2 X:boringcrypto for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0809 15:02:08.980515 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.081112 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.181235 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:08.938773 1845966 event.go:389] "Event occurred" object="kube-system/harvester-cloud-provider" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml\""
I0809 15:02:08.956101 1845966 event.go:389] "Event occurred" object="kube-system/harvester-csi-driver" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml\""
I0809 15:02:08.967675 1845966 event.go:389] "Event occurred" object="kube-system/rancher-vsphere-cpi" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml\""
I0809 15:02:08.980933 1845966 event.go:389] "Event occurred" object="kube-system/rancher-vsphere-csi" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml\""
I0809 15:02:08.999878 1845966 event.go:389] "Event occurred" object="kube-system/rke2-calico-crd" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml\""
I0809 15:02:09.014763 1845966 event.go:389] "Event occurred" object="kube-system/rke2-calico" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-calico.yaml\""
I0809 15:02:09.030373 1845966 event.go:389] "Event occurred" object="kube-system/rke2-canal" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-canal.yaml\""
I0809 15:02:09.046719 1845966 event.go:389] "Event occurred" object="kube-system/rke2-canal" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-canal.yaml\""
I0809 15:02:09.056448 1845966 leaderelection.go:250] attempting to acquire leader lease kube-system/rke2...
I0809 15:02:09.056523 1845966 leaderelection.go:250] attempting to acquire leader lease kube-system/rke2-etcd...
I0809 15:02:09.067544 1845966 event.go:389] "Event occurred" object="kube-system/rke2-cilium" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml\""
I0809 15:02:09.083292 1845966 event.go:389] "Event occurred" object="kube-system/rke2-coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml\""
I0809 15:02:09.096018 1845966 event.go:389] "Event occurred" object="kube-system/rke2-coredns" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml\""
I0809 15:02:09.102895 1845966 event.go:389] "Event occurred" object="kube-system/rke2-flannel" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-flannel.yaml\""
I0809 15:02:09.115034 1845966 event.go:389] "Event occurred" object="kube-system/rke2-ingress-nginx-config" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml\""
I0809 15:02:09.128379 1845966 event.go:389] "Event occurred" object="kube-system/rke2-ingress-nginx-config" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml\""
I0809 15:02:09.149425 1845966 event.go:389] "Event occurred" object="kube-system/rke2-ingress-nginx" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml\""
I0809 15:02:09.173452 1845966 event.go:389] "Event occurred" object="kube-system/rke2-ingress-nginx" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml\""
E0809 15:02:09.282018 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.383081 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.483945 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.584555 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.685632 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:09.740145 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="8d640a213b00d6efeaf0acb02991f1eb" podNamespace="kube-system" podName="kube-proxy-rke21"
E0809 15:02:09.786156 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:09.828642 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file1\" (UniqueName: \"kubernetes.io/host-path/8d640a213b00d6efeaf0acb02991f1eb-file1\") pod \"kube-proxy-rke21\" (UID: \"8d640a213b00d6efeaf0acb02991f1eb\") " pod="kube-system/kube-proxy-rke21"
I0809 15:02:09.828686 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file3\" (UniqueName: \"kubernetes.io/host-path/8d640a213b00d6efeaf0acb02991f1eb-file3\") pod \"kube-proxy-rke21\" (UID: \"8d640a213b00d6efeaf0acb02991f1eb\") " pod="kube-system/kube-proxy-rke21"
I0809 15:02:09.828718 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file2\" (UniqueName: \"kubernetes.io/host-path/8d640a213b00d6efeaf0acb02991f1eb-file2\") pod \"kube-proxy-rke21\" (UID: \"8d640a213b00d6efeaf0acb02991f1eb\") " pod="kube-system/kube-proxy-rke21"
I0809 15:02:09.828752 1846549 reconciler_common.go:247] "operationExecutor.VerifyControllerAttachedVolume started for volume \"file0\" (UniqueName: \"kubernetes.io/host-path/8d640a213b00d6efeaf0acb02991f1eb-file0\") pod \"kube-proxy-rke21\" (UID: \"8d640a213b00d6efeaf0acb02991f1eb\") " pod="kube-system/kube-proxy-rke21"
E0809 15:02:09.886706 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:09.987575 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.087977 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.188057 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.288173 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.389129 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.489867 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.590796 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.691606 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.792341 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.892980 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:10.993149 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.093731 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.194440 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.294957 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.395299 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.495919 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.596767 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.697802 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.798921 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.899496 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:11.999973 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.100990 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.201698 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.302779 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.403492 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.504190 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.604951 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.705936 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.806347 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:12.907403 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.008131 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.108719 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.209203 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.309627 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.409752 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.510334 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.611036 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.711941 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.812066 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:13.912744 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.013544 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.114439 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:09.350873 1845966 event.go:389] "Event occurred" object="kube-system/rke2-metrics-server" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
I0809 15:02:09.367409 1845966 event.go:389] "Event occurred" object="kube-system/rke2-metrics-server" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml\""
I0809 15:02:09.548180 1845966 event.go:389] "Event occurred" object="kube-system/rke2-multus" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-multus.yaml\""
I0809 15:02:09.769593 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller-crd" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml\""
I0809 15:02:09.782142 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller-crd" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml\""
I0809 15:02:09.953210 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml\""
I0809 15:02:09.966150 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-controller" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml\""
I0809 15:02:10.147026 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-validation-webhook" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml\""
I0809 15:02:10.160054 1845966 event.go:389] "Event occurred" object="kube-system/rke2-snapshot-validation-webhook" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml\""
E0809 15:02:14.214920 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.315237 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.347481 1846549 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:14.415545 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.516173 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.616961 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.717107 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.818033 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:14.918812 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.019680 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.120479 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.221302 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.322200 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.423364 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.523812 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.624576 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.725647 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.826272 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:15.926601 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.027169 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.127871 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.228840 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.329172 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.429633 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.530204 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.630344 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.730828 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.831684 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:16.932249 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.032911 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.133838 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.234481 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.335020 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.435667 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.536742 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.637178 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.737759 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.838806 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:17.939821 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.040536 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.141351 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.241789 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.342978 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.444166 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.545040 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.645651 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.746642 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.846867 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:18.947831 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.048870 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.149870 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:16.990861 1845966 event.go:389] "Event occurred" object="rke21" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s\": proxy error from 127.0.0.1:9345 while dialing 10.42.4.12:9443, code 502: 502 Bad Gateway"
E0809 15:02:19.250487 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.351037 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.451957 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.552883 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.652981 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.753628 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.853982 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:19.954823 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.055303 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.156120 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.256856 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.357457 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.458073 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.558913 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.659640 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.760802 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.861940 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:20.927918 1846549 resource_metrics.go:161] "Error getting summary for resourceMetric prometheus endpoint" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:20.962078 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.063067 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.163251 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.264403 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.365122 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.465281 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.565959 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.666819 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.767698 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.868223 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:21.969128 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.069877 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.170719 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.271475 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.371961 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.472802 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.572936 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.673859 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.774382 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.875427 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:22.976053 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.076906 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.177464 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.278325 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.379040 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.479991 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.580799 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.681508 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.782044 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.882419 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:23.982957 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.083548 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.183658 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:19.918742 1845966 event.go:389] "Event occurred" object="rke21" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="NodePasswordValidationComplete" message="Deferred node password secret validation complete"
E0809 15:02:24.283825 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.348401 1846549 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"rke21\" not found"
E0809 15:02:24.384744 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.485377 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.586279 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.686781 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.787214 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.887801 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:24.988580 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.089410 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.190500 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.291306 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.391696 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.492410 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.592663 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.693185 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.794325 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.894835 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:25.995389 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.096019 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.196950 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.297299 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.397424 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.497595 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.598326 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.698963 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.799395 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:26.899883 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.000887 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.101474 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.202042 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.302174 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.402562 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.503054 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.603333 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.704388 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.804576 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:27.905425 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.006340 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.106866 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.207640 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.307987 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.408437 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.509458 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.610217 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.710968 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.811793 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:28.912487 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.013116 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.114109 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.214739 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.315112 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.416148 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.516762 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.617636 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.717835 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.818555 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:29.919550 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:30.020112 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
E0809 15:02:30.120423 1846549 kubelet_node_status.go:462] "Error getting the current node from lister" err="node \"rke21\" not found"
I0809 15:02:30.323615 1846549 apiserver.go:52] "Watching apiserver"
I0809 15:02:30.335123 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="06a98001-8765-4306-8ff2-e5c5d104e360" podNamespace="kube-system" podName="rke2-canal-67pb9"
I0809 15:02:30.335390 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="4d78ab62-e2c1-4a0e-bc14-0f8d4d964548" podNamespace="cattle-monitoring-system" podName="pushprox-kube-controller-manager-client-h2ppn"
I0809 15:02:30.335493 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="f9258daf-96fd-4d3f-b2ed-a458df4228bb" podNamespace="cattle-monitoring-system" podName="rancher-monitoring-prometheus-node-exporter-l2fnl"
I0809 15:02:30.335605 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="c3172ee9-5576-4463-b059-10590f1e744b" podNamespace="cattle-monitoring-system" podName="pushprox-kube-etcd-client-zk4jm"
I0809 15:02:30.336027 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="eb800811-7140-4738-be1c-543022924686" podNamespace="cattle-monitoring-system" podName="pushprox-kube-proxy-client-r2945"
I0809 15:02:30.336150 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="1aa085cf-1222-4996-867c-34440cc1bc33" podNamespace="cattle-monitoring-system" podName="pushprox-kube-scheduler-client-kk24v"
I0809 15:02:30.336243 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="aa7414ab-15c7-4e78-8ff7-0ad16112445d" podNamespace="kube-system" podName="rke2-ingress-nginx-controller-mfhqd"
I0809 15:02:30.336372 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="901de050-3c5b-4ada-a4d4-38c607ae65de" podNamespace="cattle-provisioning-capi-system" podName="capi-controller-manager-745f484f4d-2pjnp"
I0809 15:02:30.336458 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="ddd2c993-97db-49be-8b89-5643e77075cc" podNamespace="cattle-monitoring-system" podName="pushprox-kube-etcd-proxy-6bf9dfb758-bntmr"
I0809 15:02:30.336783 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="4445ef1d-0567-416f-aa69-81c36ea28a38" podNamespace="cert-manager" podName="cert-manager-cainjector-6ff697db96-bvq9h"
I0809 15:02:30.336890 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a92080c3-2a8d-4f1c-9c9f-6c1404c2dc98" podNamespace="cengiz" podName="nginx-deployment-8564b58ddc-xbqx8"
I0809 15:02:30.337082 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a62a6cfa-93c7-48bf-807f-a485c8f52549" podNamespace="cattle-monitoring-system" podName="pushprox-kube-scheduler-proxy-8d54cf848-v49wv"
I0809 15:02:30.337182 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="a76f5364-62f3-46d8-9e01-f40863e6daf0" podNamespace="cattle-monitoring-system" podName="pushprox-kube-controller-manager-proxy-7cdd86d77d-vt5fm"
I0809 15:02:30.337355 1846549 topology_manager.go:215] "Topology Admit Handler" podUID="e0c6754a-54c2-47a9-9ee4-fb81fb80e4c8" podNamespace="cattle-logging-system" podName="rancher-logging-root-fluentbit-t2fbq"
I0809 15:02:30.390624 1846549 desired_state_of_world_populator.go:157] "Finished populating initial desired state of world"

cengizhanR · 2024-08-10T07:45:03Z

cengizhanR
Aug 10, 2024
Author

This is my config yaml.


[root@rke21 logs]# cat /etc/rancher/rke2/config.yaml
system-default-registry: 10.0.0.10:5000
secrets-encryption: true
token: fuzzybunnyslippers
tls-san:
  - rancher.exemple.com
node-ip: "10.0.0.2"
etcd-expose-metrics: true
write-kubeconfig-mode: "0600"
use-service-account-credentials: true
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-proxy-arg:
- metrics-bind-address=0.0.0.0:10249
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- audit-policy-file=/etc/rancher/rke2/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
- audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- "eviction-hard=imagefs.available<5%,memory.available<1Gi,nodefs.available<5%"
- "eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=10%,imagefs.available=10%"
- "kube-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi"
- "system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi"
- "oom-score-adj=-999"
- "image-gc-high-threshold=85"
- "image-gc-low-threshold=80"

0 replies

brandond · 2024-08-14T06:21:23Z

brandond
Aug 14, 2024
Maintainer

What about rke2-server, apiserver, and etcd pod logs? The kubelet logs don't contain much useful information.

6 replies

brandond Aug 14, 2024
Maintainer

The logs indicate that the service was intentionally stopped, by something or someone. It did not crash.

You can find pod logs in /var/log/pods even if the kubelet is down.

cengizhanR Aug 14, 2024
Author

@brandond But I saw that the rke2-server status was failed. If one of us had stopped it, shouldn't it have been stopped?
This is kube-api-server logs,
These are the kube-api-serverlogs from the day of the crash. Thank you very much.

2024-08-09T15:01:59.551718331+03:00 stderr F I0809 12:01:59.551555       1 options.go:221] external host was not specified, using 10.0.0.2
2024-08-09T15:01:59.552677046+03:00 stderr F I0809 12:01:59.552571       1 server.go:148] Version: v1.30.0+rke2r1
2024-08-09T15:01:59.552694746+03:00 stderr F I0809 12:01:59.552606       1 server.go:150] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
2024-08-09T15:01:59.7302826+03:00 stderr F I0809 12:01:59.730063       1 shared_informer.go:313] Waiting for caches to sync for node_authorizer
2024-08-09T15:01:59.736394195+03:00 stderr F I0809 12:01:59.736286       1 shared_informer.go:313] Waiting for caches to sync for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]
2024-08-09T15:01:59.739200239+03:00 stderr F I0809 12:01:59.739121       1 plugins.go:157] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
2024-08-09T15:01:59.739214439+03:00 stderr F I0809 12:01:59.739143       1 plugins.go:160] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
2024-08-09T15:01:59.739454443+03:00 stderr F I0809 12:01:59.739388       1 instance.go:299] Using reconciler: lease
2024-08-09T15:01:59.761706488+03:00 stderr F I0809 12:01:59.761602       1 handler.go:286] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager
2024-08-09T15:01:59.761726288+03:00 stderr F W0809 12:01:59.761630       1 genericapiserver.go:733] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:01:59.889959677+03:00 stderr F I0809 12:01:59.889825       1 handler.go:286] Adding GroupVersion  v1 to ResourceManager
2024-08-09T15:01:59.89015268+03:00 stderr F I0809 12:01:59.890073       1 instance.go:696] API group "internal.apiserver.k8s.io" is not enabled, skipping.
2024-08-09T15:02:00.010963054+03:00 stderr F I0809 12:02:00.010840       1 instance.go:696] API group "storagemigration.k8s.io" is not enabled, skipping.
2024-08-09T15:02:00.088648059+03:00 stderr F I0809 12:02:00.088516       1 instance.go:696] API group "resource.k8s.io" is not enabled, skipping.
2024-08-09T15:02:00.112023021+03:00 stderr F I0809 12:02:00.111614       1 handler.go:286] Adding GroupVersion authentication.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.112048622+03:00 stderr F W0809 12:02:00.111645       1 genericapiserver.go:733] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.112051922+03:00 stderr F W0809 12:02:00.111697       1 genericapiserver.go:733] Skipping API authentication.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.11322884+03:00 stderr F I0809 12:02:00.113146       1 handler.go:286] Adding GroupVersion authorization.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.11323864+03:00 stderr F W0809 12:02:00.113170       1 genericapiserver.go:733] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.114610662+03:00 stderr F I0809 12:02:00.114541       1 handler.go:286] Adding GroupVersion autoscaling v2 to ResourceManager
2024-08-09T15:02:00.11582578+03:00 stderr F I0809 12:02:00.115760       1 handler.go:286] Adding GroupVersion autoscaling v1 to ResourceManager
2024-08-09T15:02:00.115832981+03:00 stderr F W0809 12:02:00.115779       1 genericapiserver.go:733] Skipping API autoscaling/v2beta1 because it has no resources.
2024-08-09T15:02:00.115834781+03:00 stderr F W0809 12:02:00.115784       1 genericapiserver.go:733] Skipping API autoscaling/v2beta2 because it has no resources.
2024-08-09T15:02:00.117150301+03:00 stderr F I0809 12:02:00.117085       1 handler.go:286] Adding GroupVersion batch v1 to ResourceManager
2024-08-09T15:02:00.117157401+03:00 stderr F W0809 12:02:00.117103       1 genericapiserver.go:733] Skipping API batch/v1beta1 because it has no resources.
2024-08-09T15:02:00.118097716+03:00 stderr F I0809 12:02:00.118024       1 handler.go:286] Adding GroupVersion certificates.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.118104916+03:00 stderr F W0809 12:02:00.118040       1 genericapiserver.go:733] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.118107116+03:00 stderr F W0809 12:02:00.118044       1 genericapiserver.go:733] Skipping API certificates.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.118575923+03:00 stderr F I0809 12:02:00.118522       1 handler.go:286] Adding GroupVersion coordination.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.118583123+03:00 stderr F W0809 12:02:00.118534       1 genericapiserver.go:733] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.118670625+03:00 stderr F W0809 12:02:00.118579       1 genericapiserver.go:733] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.119143132+03:00 stderr F I0809 12:02:00.119084       1 handler.go:286] Adding GroupVersion discovery.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.120780557+03:00 stderr F I0809 12:02:00.120658       1 handler.go:286] Adding GroupVersion networking.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.120789757+03:00 stderr F W0809 12:02:00.120676       1 genericapiserver.go:733] Skipping API networking.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.120792157+03:00 stderr F W0809 12:02:00.120681       1 genericapiserver.go:733] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.121177663+03:00 stderr F I0809 12:02:00.121115       1 handler.go:286] Adding GroupVersion node.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.121181763+03:00 stderr F W0809 12:02:00.121129       1 genericapiserver.go:733] Skipping API node.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.121184164+03:00 stderr F W0809 12:02:00.121133       1 genericapiserver.go:733] Skipping API node.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.121969276+03:00 stderr F I0809 12:02:00.121911       1 handler.go:286] Adding GroupVersion policy v1 to ResourceManager
2024-08-09T15:02:00.121974776+03:00 stderr F W0809 12:02:00.121926       1 genericapiserver.go:733] Skipping API policy/v1beta1 because it has no resources.
2024-08-09T15:02:00.123612001+03:00 stderr F I0809 12:02:00.123549       1 handler.go:286] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.123618501+03:00 stderr F W0809 12:02:00.123565       1 genericapiserver.go:733] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.123625001+03:00 stderr F W0809 12:02:00.123570       1 genericapiserver.go:733] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.124245211+03:00 stderr F I0809 12:02:00.124098       1 handler.go:286] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.124249611+03:00 stderr F W0809 12:02:00.124115       1 genericapiserver.go:733] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.124259211+03:00 stderr F W0809 12:02:00.124119       1 genericapiserver.go:733] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.126480046+03:00 stderr F I0809 12:02:00.126409       1 handler.go:286] Adding GroupVersion storage.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.126490046+03:00 stderr F W0809 12:02:00.126426       1 genericapiserver.go:733] Skipping API storage.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.126492946+03:00 stderr F W0809 12:02:00.126431       1 genericapiserver.go:733] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.127681264+03:00 stderr F I0809 12:02:00.127614       1 handler.go:286] Adding GroupVersion flowcontrol.apiserver.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.12871288+03:00 stderr F I0809 12:02:00.128647       1 handler.go:286] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta3 to ResourceManager
2024-08-09T15:02:00.12872228+03:00 stderr F W0809 12:02:00.128664       1 genericapiserver.go:733] Skipping API flowcontrol.apiserver.k8s.io/v1beta2 because it has no resources.
2024-08-09T15:02:00.12872498+03:00 stderr F W0809 12:02:00.128669       1 genericapiserver.go:733] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.132242935+03:00 stderr F I0809 12:02:00.132171       1 handler.go:286] Adding GroupVersion apps v1 to ResourceManager
2024-08-09T15:02:00.132264835+03:00 stderr F W0809 12:02:00.132189       1 genericapiserver.go:733] Skipping API apps/v1beta2 because it has no resources.
2024-08-09T15:02:00.132268835+03:00 stderr F W0809 12:02:00.132194       1 genericapiserver.go:733] Skipping API apps/v1beta1 because it has no resources.
2024-08-09T15:02:00.134398768+03:00 stderr F I0809 12:02:00.134334       1 handler.go:286] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.134404969+03:00 stderr F W0809 12:02:00.134350       1 genericapiserver.go:733] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.134406569+03:00 stderr F W0809 12:02:00.134355       1 genericapiserver.go:733] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
2024-08-09T15:02:00.135341183+03:00 stderr F I0809 12:02:00.135283       1 handler.go:286] Adding GroupVersion events.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.135346283+03:00 stderr F W0809 12:02:00.135301       1 genericapiserver.go:733] Skipping API events.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.144736329+03:00 stderr F I0809 12:02:00.144648       1 handler.go:286] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.144756329+03:00 stderr F W0809 12:02:00.144673       1 genericapiserver.go:733] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
2024-08-09T15:02:00.598167562+03:00 stderr F I0809 12:02:00.598032       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/rke2/server/tls/request-header-ca.crt"
2024-08-09T15:02:00.598278164+03:00 stderr F I0809 12:02:00.598095       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/rke2/server/tls/client-ca.crt"
2024-08-09T15:02:00.598297964+03:00 stderr F I0809 12:02:00.598201       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key"
2024-08-09T15:02:00.598569568+03:00 stderr F I0809 12:02:00.598501       1 secure_serving.go:213] Serving securely on [::]:6443
2024-08-09T15:02:00.598579068+03:00 stderr F I0809 12:02:00.598532       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
2024-08-09T15:02:00.602387527+03:00 stderr F I0809 12:02:00.598659       1 system_namespaces_controller.go:67] Starting system namespaces controller
2024-08-09T15:02:00.602417228+03:00 stderr F I0809 12:02:00.598839       1 available_controller.go:423] Starting AvailableConditionController
2024-08-09T15:02:00.602431228+03:00 stderr F I0809 12:02:00.598853       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
2024-08-09T15:02:00.602435128+03:00 stderr F I0809 12:02:00.598903       1 gc_controller.go:78] Starting apiserver lease garbage collector
2024-08-09T15:02:00.602438628+03:00 stderr F I0809 12:02:00.598948       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
2024-08-09T15:02:00.602451628+03:00 stderr F I0809 12:02:00.598954       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
2024-08-09T15:02:00.602453828+03:00 stderr F I0809 12:02:00.599286       1 controller.go:78] Starting OpenAPI AggregationController
2024-08-09T15:02:00.602456028+03:00 stderr F I0809 12:02:00.599358       1 controller.go:94] "Starting controller" name="encryption-provider-config-automatic-reload-controller"
2024-08-09T15:02:00.602458028+03:00 stderr F I0809 12:02:00.599483       1 apf_controller.go:374] Starting API Priority and Fairness config controller
2024-08-09T15:02:00.602459928+03:00 stderr F I0809 12:02:00.599515       1 controller.go:80] Starting OpenAPI V3 AggregationController
2024-08-09T15:02:00.602462128+03:00 stderr F I0809 12:02:00.599516       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt::/var/lib/rancher/rke2/server/tls/client-auth-proxy.key"
2024-08-09T15:02:00.602464728+03:00 stderr F I0809 12:02:00.599563       1 aggregator.go:163] waiting for initial CRD sync...
2024-08-09T15:02:00.602466629+03:00 stderr F I0809 12:02:00.599286       1 controller.go:116] Starting legacy_token_tracking_controller
2024-08-09T15:02:00.602468429+03:00 stderr F I0809 12:02:00.599584       1 shared_informer.go:313] Waiting for caches to sync for configmaps
2024-08-09T15:02:00.602470429+03:00 stderr F I0809 12:02:00.600441       1 customresource_discovery_controller.go:289] Starting DiscoveryController
2024-08-09T15:02:00.602472529+03:00 stderr F I0809 12:02:00.600689       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
2024-08-09T15:02:00.602474429+03:00 stderr F I0809 12:02:00.600700       1 shared_informer.go:313] Waiting for caches to sync for cluster_authentication_trust_controller
2024-08-09T15:02:00.602476229+03:00 stderr F I0809 12:02:00.600749       1 crdregistration_controller.go:111] Starting crd-autoregister controller
2024-08-09T15:02:00.602478329+03:00 stderr F I0809 12:02:00.600758       1 shared_informer.go:313] Waiting for caches to sync for crd-autoregister
2024-08-09T15:02:00.602480029+03:00 stderr F I0809 12:02:00.600802       1 controller.go:139] Starting OpenAPI controller
2024-08-09T15:02:00.602481829+03:00 stderr F I0809 12:02:00.600840       1 controller.go:87] Starting OpenAPI V3 controller
2024-08-09T15:02:00.602483629+03:00 stderr F I0809 12:02:00.600866       1 naming_controller.go:291] Starting NamingConditionController
2024-08-09T15:02:00.602485729+03:00 stderr F I0809 12:02:00.600887       1 establishing_controller.go:76] Starting EstablishingController
2024-08-09T15:02:00.602488529+03:00 stderr F I0809 12:02:00.600907       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
2024-08-09T15:02:00.602491329+03:00 stderr F I0809 12:02:00.600925       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
2024-08-09T15:02:00.602494529+03:00 stderr F I0809 12:02:00.600943       1 crd_finalizer.go:266] Starting CRDFinalizer
2024-08-09T15:02:00.602497529+03:00 stderr F I0809 12:02:00.601058       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/rke2/server/tls/client-ca.crt"
2024-08-09T15:02:00.602500529+03:00 stderr F I0809 12:02:00.601164       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/rke2/server/tls/request-header-ca.crt"
2024-08-09T15:02:00.637903678+03:00 stderr F I0809 12:02:00.637600       1 shared_informer.go:320] Caches are synced for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]
2024-08-09T15:02:00.637931879+03:00 stderr F I0809 12:02:00.637621       1 policy_source.go:224] refreshing policies
2024-08-09T15:02:00.699416432+03:00 stderr F I0809 12:02:00.699100       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
2024-08-09T15:02:00.699446533+03:00 stderr F I0809 12:02:00.699141       1 cache.go:39] Caches are synced for AvailableConditionController controller
2024-08-09T15:02:00.699674036+03:00 stderr F I0809 12:02:00.699586       1 apf_controller.go:379] Running API Priority and Fairness config worker
2024-08-09T15:02:00.699738437+03:00 stderr F I0809 12:02:00.699700       1 apf_controller.go:382] Running API Priority and Fairness periodic rebalancing process
2024-08-09T15:02:00.699861939+03:00 stderr F I0809 12:02:00.699633       1 shared_informer.go:320] Caches are synced for configmaps
2024-08-09T15:02:00.700937556+03:00 stderr F I0809 12:02:00.700863       1 shared_informer.go:320] Caches are synced for cluster_authentication_trust_controller
2024-08-09T15:02:00.72696906+03:00 stderr F I0809 12:02:00.726842       1 handler_discovery.go:447] Starting ResourceDiscoveryManager
2024-08-09T15:02:00.73084642+03:00 stderr F I0809 12:02:00.730740       1 shared_informer.go:320] Caches are synced for node_authorizer
2024-08-09T15:02:00.744748835+03:00 stderr F I0809 12:02:00.744512       1 controller.go:615] quota admission added evaluator for: leases.coordination.k8s.io
2024-08-09T15:02:00.90107026+03:00 stderr F I0809 12:02:00.900923       1 shared_informer.go:320] Caches are synced for crd-autoregister
2024-08-09T15:02:00.901505567+03:00 stderr F I0809 12:02:00.901269       1 handler.go:286] Adding GroupVersion management.cattle.io v3 to ResourceManager
2024-08-09T15:02:00.901824272+03:00 stderr F I0809 12:02:00.901768       1 handler.go:286] Adding GroupVersion crd.projectcalico.org v1 to ResourceManager
2024-08-09T15:02:00.901885373+03:00 stderr F I0809 12:02:00.901847       1 handler.go:286] Adding GroupVersion cluster.x-k8s.io v1alpha3 to ResourceManager
2024-08-09T15:02:00.901940674+03:00 stderr F I0809 12:02:00.901906       1 handler.go:286] Adding GroupVersion cluster.x-k8s.io v1alpha4 to ResourceManager
2024-08-09T15:02:00.902020275+03:00 stderr F I0809 12:02:00.901959       1 handler.go:286] Adding GroupVersion cluster.x-k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:00.902153777+03:00 stderr F I0809 12:02:00.902111       1 handler.go:286] Adding GroupVersion rke-machine.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.902184277+03:00 stderr F I0809 12:02:00.902154       1 handler.go:286] Adding GroupVersion acme.cert-manager.io v1 to ResourceManager
2024-08-09T15:02:00.902650385+03:00 stderr F I0809 12:02:00.902575       1 handler.go:286] Adding GroupVersion fleet.cattle.io v1alpha1 to ResourceManager
2024-08-09T15:02:00.902779087+03:00 stderr F I0809 12:02:00.902729       1 handler.go:286] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager
2024-08-09T15:02:00.91591929+03:00 stderr F I0809 12:02:00.903427       1 handler.go:286] Adding GroupVersion rke.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.915944391+03:00 stderr F I0809 12:02:00.903829       1 aggregator.go:165] initial CRD sync complete...
2024-08-09T15:02:00.915947691+03:00 stderr F I0809 12:02:00.903844       1 autoregister_controller.go:141] Starting autoregister controller
2024-08-09T15:02:00.915950391+03:00 stderr F I0809 12:02:00.903852       1 cache.go:32] Waiting for caches to sync for autoregister controller
2024-08-09T15:02:00.915953091+03:00 stderr F I0809 12:02:00.903859       1 cache.go:39] Caches are synced for autoregister controller
2024-08-09T15:02:00.915955391+03:00 stderr F I0809 12:02:00.904489       1 handler.go:286] Adding GroupVersion catalog.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.915958991+03:00 stderr F I0809 12:02:00.904545       1 handler.go:286] Adding GroupVersion monitoring.coreos.com v1alpha1 to ResourceManager
2024-08-09T15:02:00.915961191+03:00 stderr F I0809 12:02:00.906152       1 handler.go:286] Adding GroupVersion provisioning.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.915963391+03:00 stderr F I0809 12:02:00.906470       1 handler.go:286] Adding GroupVersion dapr.io v1alpha1 to ResourceManager
2024-08-09T15:02:00.915986891+03:00 stderr F I0809 12:02:00.906506       1 handler.go:286] Adding GroupVersion dapr.io v2alpha1 to ResourceManager
2024-08-09T15:02:00.915990191+03:00 stderr F I0809 12:02:00.906535       1 handler.go:286] Adding GroupVersion project.cattle.io v3 to ResourceManager
2024-08-09T15:02:00.916008292+03:00 stderr F I0809 12:02:00.907446       1 handler.go:286] Adding GroupVersion rke-machine-config.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.916010892+03:00 stderr F I0809 12:02:00.910328       1 handler.go:286] Adding GroupVersion snapshot.storage.k8s.io v1 to ResourceManager
2024-08-09T15:02:00.916013492+03:00 stderr F I0809 12:02:00.910371       1 handler.go:286] Adding GroupVersion snapshot.storage.k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:00.916015392+03:00 stderr F I0809 12:02:00.910502       1 handler.go:286] Adding GroupVersion cert-manager.io v1 to ResourceManager
2024-08-09T15:02:00.916017592+03:00 stderr F I0809 12:02:00.910846       1 handler.go:286] Adding GroupVersion logging-extensions.banzaicloud.io v1alpha1 to ResourceManager
2024-08-09T15:02:00.916344897+03:00 stderr F I0809 12:02:00.913968       1 handler.go:286] Adding GroupVersion logging.banzaicloud.io v1alpha1 to ResourceManager
2024-08-09T15:02:00.916354697+03:00 stderr F I0809 12:02:00.914026       1 handler.go:286] Adding GroupVersion logging.banzaicloud.io v1beta1 to ResourceManager
2024-08-09T15:02:00.916357897+03:00 stderr F I0809 12:02:00.914822       1 handler.go:286] Adding GroupVersion ui.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.916359997+03:00 stderr F I0809 12:02:00.915874       1 handler.go:286] Adding GroupVersion gitjob.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.917619417+03:00 stderr F I0809 12:02:00.917525       1 handler.go:286] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
2024-08-09T15:02:00.918406529+03:00 stderr F I0809 12:02:00.918347       1 handler.go:286] Adding GroupVersion helm.cattle.io v1 to ResourceManager
2024-08-09T15:02:01.628523643+03:00 stderr F I0809 12:02:01.628338       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
2024-08-09T15:02:05.711429672+03:00 stderr F E0809 12:02:05.711289       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:05.711456572+03:00 stderr F E0809 12:02:05.711296       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1": context deadline exceeded
2024-08-09T15:02:05.727897427+03:00 stderr F I0809 12:02:05.727778       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:05.727920628+03:00 stderr F I0809 12:02:05.727809       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:05.815610788+03:00 stderr F W0809 12:02:05.815501       1 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.0.0.2 10.0.0.3 10.0.0.4]
2024-08-09T15:02:05.817007709+03:00 stderr F I0809 12:02:05.816774       1 controller.go:615] quota admission added evaluator for: endpoints
2024-08-09T15:02:05.825704644+03:00 stderr F I0809 12:02:05.825486       1 controller.go:615] quota admission added evaluator for: endpointslices.discovery.k8s.io
2024-08-09T15:02:07.064756863+03:00 stderr F I0809 12:02:07.064629       1 trace.go:236] Trace[1775223652]: "List" accept:application/json, */*,audit-id:9929278c-15d3-4530-8d72-a09af0edd76a,client:127.0.0.1,api-group:apiextensions.k8s.io,api-version:v1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:customresourcedefinitions,scope:cluster,url:/apis/apiextensions.k8s.io/v1/customresourcedefinitions,user-agent:rke2-supervisor@rke21/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:LIST (09-Aug-2024 12:02:06.539) (total time: 524ms):
2024-08-09T15:02:07.064787263+03:00 stderr F Trace[1775223652]: ---"Writing http response done" count:161 208ms (12:02:07.064)
2024-08-09T15:02:07.064790463+03:00 stderr F Trace[1775223652]: [524.884341ms] [524.884341ms] END
2024-08-09T15:02:07.646573887+03:00 stderr F I0809 12:02:07.646420       1 controller.go:615] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
2024-08-09T15:02:07.838124558+03:00 stderr F I0809 12:02:07.838001       1 trace.go:236] Trace[1818682204]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.700) (total time: 7137ms):
2024-08-09T15:02:07.838154259+03:00 stderr F Trace[1818682204]: [7.137938214s] [7.137938214s] END
2024-08-09T15:02:07.838359762+03:00 stderr F I0809 12:02:07.838171       1 trace.go:236] Trace[1073843335]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.700) (total time: 7137ms):
2024-08-09T15:02:07.838377962+03:00 stderr F Trace[1073843335]: [7.137976915s] [7.137976915s] END
2024-08-09T15:02:07.838381162+03:00 stderr F I0809 12:02:07.838189       1 trace.go:236] Trace[1470687871]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.727) (total time: 7111ms):
2024-08-09T15:02:07.838383562+03:00 stderr F Trace[1470687871]: [7.111124498s] [7.111124498s] END
2024-08-09T15:02:07.838386862+03:00 stderr F I0809 12:02:07.838207       1 trace.go:236] Trace[142988027]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.700) (total time: 7137ms):
2024-08-09T15:02:07.838389262+03:00 stderr F Trace[142988027]: [7.137764711s] [7.137764711s] END
2024-08-09T15:02:07.838391662+03:00 stderr F I0809 12:02:07.838212       1 trace.go:236] Trace[576485209]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.727) (total time: 7111ms):
2024-08-09T15:02:07.838393762+03:00 stderr F Trace[576485209]: [7.111152198s] [7.111152198s] END
2024-08-09T15:02:07.838395962+03:00 stderr F I0809 12:02:07.838246       1 trace.go:236] Trace[121526346]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.838398062+03:00 stderr F Trace[121526346]: [7.138202718s] [7.138202718s] END
2024-08-09T15:02:07.838400162+03:00 stderr F I0809 12:02:07.838278       1 trace.go:236] Trace[1295934119]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.838410963+03:00 stderr F Trace[1295934119]: [7.138065916s] [7.138065916s] END
2024-08-09T15:02:07.838490864+03:00 stderr F E0809 12:02:07.838363       1 controller.go:146] Error updating APIService "v1beta1.custom.metrics.k8s.io" with err: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.1.4:6443, code 502: 502 Bad Gateway
2024-08-09T15:02:07.838500664+03:00 stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-08-09T15:02:07.838668067+03:00 stderr F I0809 12:02:07.838554       1 trace.go:236] Trace[1847676887]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.699) (total time: 7138ms):
2024-08-09T15:02:07.838674667+03:00 stderr F Trace[1847676887]: [7.138550124s] [7.138550124s] END
2024-08-09T15:02:07.838709467+03:00 stderr F I0809 12:02:07.838665       1 trace.go:236] Trace[1360772068]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.727) (total time: 7111ms):
2024-08-09T15:02:07.838717167+03:00 stderr F Trace[1360772068]: [7.111608805s] [7.111608805s] END
2024-08-09T15:02:07.838787068+03:00 stderr F I0809 12:02:07.838742       1 trace.go:236] Trace[2102264967]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.838802869+03:00 stderr F Trace[2102264967]: [7.138158118s] [7.138158118s] END
2024-08-09T15:02:07.839573881+03:00 stderr F I0809 12:02:07.838836       1 trace.go:236] Trace[1004597310]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.839585081+03:00 stderr F Trace[1004597310]: [7.138241019s] [7.138241019s] END
2024-08-09T15:02:07.839586881+03:00 stderr F I0809 12:02:07.839182       1 trace.go:236] Trace[1118707448]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.839588381+03:00 stderr F Trace[1118707448]: [7.138702726s] [7.138702726s] END
2024-08-09T15:02:07.839590181+03:00 stderr F I0809 12:02:07.839330       1 trace.go:236] Trace[1516279824]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:00.700) (total time: 7138ms):
2024-08-09T15:02:07.839592081+03:00 stderr F Trace[1516279824]: [7.138407822s] [7.138407822s] END
2024-08-09T15:02:08.92589613+03:00 stderr F I0809 12:02:08.925731       1 trace.go:236] Trace[1794976874]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:01.833) (total time: 7092ms):
2024-08-09T15:02:08.925922831+03:00 stderr F Trace[1794976874]: [7.092481509s] [7.092481509s] END
2024-08-09T15:02:08.925926831+03:00 stderr F E0809 12:02:08.925801       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-08-09T15:02:08.925934331+03:00 stderr F I0809 12:02:08.925815       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-08-09T15:02:09.048871138+03:00 stderr F I0809 12:02:09.048579       1 controller.go:615] quota admission added evaluator for: addons.k3s.cattle.io
2024-08-09T15:02:10.713420356+03:00 stderr F E0809 12:02:10.712998       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:10.719355648+03:00 stderr F E0809 12:02:10.718350       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:12.830033986+03:00 stderr F I0809 12:02:12.829873       1 trace.go:236] Trace[767664637]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.712) (total time: 7116ms):
2024-08-09T15:02:12.830059486+03:00 stderr F Trace[767664637]: [7.116904088s] [7.116904088s] END
2024-08-09T15:02:12.830075287+03:00 stderr F I0809 12:02:12.829987       1 trace.go:236] Trace[921043238]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.711) (total time: 7118ms):
2024-08-09T15:02:12.830079687+03:00 stderr F Trace[921043238]: [7.118506813s] [7.118506813s] END
2024-08-09T15:02:12.830087987+03:00 stderr F I0809 12:02:12.830028       1 trace.go:236] Trace[238130808]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.713) (total time: 7116ms):
2024-08-09T15:02:12.830091287+03:00 stderr F Trace[238130808]: [7.116936188s] [7.116936188s] END
2024-08-09T15:02:12.830249189+03:00 stderr F I0809 12:02:12.830153       1 trace.go:236] Trace[1327616654]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.711) (total time: 7118ms):
2024-08-09T15:02:12.83029289+03:00 stderr F Trace[1327616654]: [7.118715615s] [7.118715615s] END
2024-08-09T15:02:12.830732997+03:00 stderr F I0809 12:02:12.830366       1 trace.go:236] Trace[665023580]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.713) (total time: 7117ms):
2024-08-09T15:02:12.830744297+03:00 stderr F Trace[665023580]: [7.117252393s] [7.117252393s] END
2024-08-09T15:02:12.830748597+03:00 stderr F I0809 12:02:12.830407       1 trace.go:236] Trace[254290950]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.711) (total time: 7118ms):
2024-08-09T15:02:12.830751897+03:00 stderr F Trace[254290950]: [7.118867718s] [7.118867718s] END
2024-08-09T15:02:12.830755097+03:00 stderr F I0809 12:02:12.830514       1 trace.go:236] Trace[1412960907]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.728) (total time: 7102ms):
2024-08-09T15:02:12.830757897+03:00 stderr F Trace[1412960907]: [7.102475364s] [7.102475364s] END
2024-08-09T15:02:12.830759997+03:00 stderr F I0809 12:02:12.830523       1 trace.go:236] Trace[1315461899]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.712) (total time: 7117ms):
2024-08-09T15:02:12.830761797+03:00 stderr F Trace[1315461899]: [7.117559998s] [7.117559998s] END
2024-08-09T15:02:12.830763797+03:00 stderr F I0809 12:02:12.830639       1 trace.go:236] Trace[1207485289]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.711) (total time: 7119ms):
2024-08-09T15:02:12.830765597+03:00 stderr F Trace[1207485289]: [7.119138122s] [7.119138122s] END
2024-08-09T15:02:12.830767797+03:00 stderr F I0809 12:02:12.830648       1 trace.go:236] Trace[1142738377]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.727) (total time: 7102ms):
2024-08-09T15:02:12.830770497+03:00 stderr F Trace[1142738377]: [7.102642066s] [7.102642066s] END
2024-08-09T15:02:12.830813598+03:00 stderr F I0809 12:02:12.830774       1 trace.go:236] Trace[1175308806]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:05.711) (total time: 7119ms):
2024-08-09T15:02:12.830817298+03:00 stderr F Trace[1175308806]: [7.119316325s] [7.119316325s] END
2024-08-09T15:02:12.830876699+03:00 stderr F I0809 12:02:12.830814       1 trace.go:236] Trace[1057539985]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:05.713) (total time: 7117ms):
2024-08-09T15:02:12.830887299+03:00 stderr F Trace[1057539985]: [7.117623499s] [7.117623499s] END
2024-08-09T15:02:14.942090845+03:00 stderr F I0809 12:02:14.941874       1 trace.go:236] Trace[1377664785]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:07.838) (total time: 7103ms):
2024-08-09T15:02:14.942114946+03:00 stderr F Trace[1377664785]: [7.103341177s] [7.103341177s] END
2024-08-09T15:02:14.942120946+03:00 stderr F E0809 12:02:14.942010       1 controller.go:102] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.1.4:6443, code 502: 502 Bad Gateway
2024-08-09T15:02:14.942125246+03:00 stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-08-09T15:02:14.942129446+03:00 stderr F I0809 12:02:14.942026       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-08-09T15:02:15.722141644+03:00 stderr F E0809 12:02:15.721982       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:15.726127006+03:00 stderr F E0809 12:02:15.725978       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:16.029934818+03:00 stderr F I0809 12:02:16.029780       1 trace.go:236] Trace[117822910]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:08.926) (total time: 7103ms):
2024-08-09T15:02:16.029960319+03:00 stderr F Trace[117822910]: [7.103671782s] [7.103671782s] END
2024-08-09T15:02:16.029974519+03:00 stderr F E0809 12:02:16.029906       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-08-09T15:02:16.029978219+03:00 stderr F I0809 12:02:16.029951       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-08-09T15:02:16.670205249+03:00 stderr F W0809 12:02:16.670084       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.12:9443, code 502: 502 Bad Gateway
2024-08-09T15:02:16.670556255+03:00 stderr F I0809 12:02:16.670330       1 trace.go:236] Trace[1303379267]: "Create" accept:application/json, */*,audit-id:a10833fe-0b36-4c61-a6fb-ad32372e5137,client:127.0.0.1,api-group:,api-version:v1,name:,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/kube-system/secrets,user-agent:rke2-supervisor@rke21/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:POST (09-Aug-2024 12:02:09.540) (total time: 7129ms):
2024-08-09T15:02:16.670572655+03:00 stderr F Trace[1303379267]: ["Call mutating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.secrets,resource:/v1, Resource=secrets,subresource:,operation:CREATE,UID:6cf001e2-c6e8-4a7a-a31e-ea6f96a21531 7128ms (12:02:09.541)
2024-08-09T15:02:16.670576855+03:00 stderr F Trace[1303379267]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.12:9443 7128ms (12:02:09.541)]]
2024-08-09T15:02:16.670580955+03:00 stderr F Trace[1303379267]: [7.129855289s] [7.129855289s] END
2024-08-09T15:02:16.990087211+03:00 stderr F W0809 12:02:16.989939       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.12:9443, code 502: 502 Bad Gateway
2024-08-09T15:02:16.990229113+03:00 stderr F I0809 12:02:16.990156       1 trace.go:236] Trace[696419277]: "Create" accept:application/json, */*,audit-id:d689bafe-bf61-4b99-9ff9-84cb762ee9c8,client:127.0.0.1,api-group:,api-version:v1,name:,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/kube-system/secrets,user-agent:rke2-supervisor@rke21/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:POST (09-Aug-2024 12:02:09.895) (total time: 7094ms):
2024-08-09T15:02:16.990237413+03:00 stderr F Trace[696419277]: ["Call mutating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.secrets,resource:/v1, Resource=secrets,subresource:,operation:CREATE,UID:2704acf2-4956-49a8-973a-97a06259f828 7094ms (12:02:09.896)
2024-08-09T15:02:16.990242413+03:00 stderr F Trace[696419277]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.12:9443 7093ms (12:02:09.896)]]
2024-08-09T15:02:16.990245713+03:00 stderr F Trace[696419277]: [7.094387738s] [7.094387738s] END
2024-08-09T15:02:17.822052615+03:00 stderr F I0809 12:02:17.821908       1 trace.go:236] Trace[397278977]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:10.713) (total time: 7108ms):
2024-08-09T15:02:17.822098116+03:00 stderr F Trace[397278977]: [7.108470657s] [7.108470657s] END
2024-08-09T15:02:17.822103416+03:00 stderr F I0809 12:02:17.821962       1 trace.go:236] Trace[1534487253]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:10.713) (total time: 7108ms):
2024-08-09T15:02:17.822106716+03:00 stderr F Trace[1534487253]: [7.108509757s] [7.108509757s] END
2024-08-09T15:02:17.822126516+03:00 stderr F I0809 12:02:17.822034       1 trace.go:236] Trace[2025626861]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:10.713) (total time: 7108ms):
2024-08-09T15:02:17.822130416+03:00 stderr F Trace[2025626861]: [7.10861906s] [7.10861906s] END
2024-08-09T15:02:17.822133317+03:00 stderr F I0809 12:02:17.822033       1 trace.go:236] Trace[463912371]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:10.718) (total time: 7103ms):
2024-08-09T15:02:17.822136017+03:00 stderr F Trace[463912371]: [7.10350598s] [7.10350598s] END
2024-08-09T15:02:17.822306919+03:00 stderr F I0809 12:02:17.822205       1 trace.go:236] Trace[1656400812]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:10.718) (total time: 7103ms):
2024-08-09T15:02:17.82232682+03:00 stderr F Trace[1656400812]: [7.10352898s] [7.10352898s] END
2024-08-09T15:02:17.82233712+03:00 stderr F I0809 12:02:17.822273       1 trace.go:236] Trace[875502046]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:10.718) (total time: 7103ms):
2024-08-09T15:02:17.82233962+03:00 stderr F Trace[875502046]: [7.10354218s] [7.10354218s] END
2024-08-09T15:02:17.822523123+03:00 stderr F I0809 12:02:17.822420       1 trace.go:236] Trace[270828406]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:10.713) (total time: 7108ms):
2024-08-09T15:02:17.822533323+03:00 stderr F Trace[270828406]: [7.108888764s] [7.108888764s] END
2024-08-09T15:02:17.822536523+03:00 stderr F I0809 12:02:17.822441       1 trace.go:236] Trace[782530444]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:10.713) (total time: 7108ms):
2024-08-09T15:02:17.822539223+03:00 stderr F Trace[782530444]: [7.108992265s] [7.108992265s] END
2024-08-09T15:02:17.822757526+03:00 stderr F I0809 12:02:17.822682       1 trace.go:236] Trace[2013108571]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:10.718) (total time: 7104ms):
2024-08-09T15:02:17.822763726+03:00 stderr F Trace[2013108571]: [7.104060789s] [7.104060789s] END
2024-08-09T15:02:17.822906629+03:00 stderr F I0809 12:02:17.822839       1 trace.go:236] Trace[1727771959]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:10.718) (total time: 7103ms):
2024-08-09T15:02:17.822913329+03:00 stderr F Trace[1727771959]: [7.103940686s] [7.103940686s] END
2024-08-09T15:02:20.727284977+03:00 stderr F E0809 12:02:20.727130       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:20.730881333+03:00 stderr F E0809 12:02:20.730785       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:22.046013332+03:00 stderr F I0809 12:02:22.045875       1 trace.go:236] Trace[1954895185]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:14.942) (total time: 7103ms):
2024-08-09T15:02:22.046036332+03:00 stderr F Trace[1954895185]: [7.103666182s] [7.103666182s] END
2024-08-09T15:02:22.046057332+03:00 stderr F E0809 12:02:22.046002       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.3.2:10250, code 502: 502 Bad Gateway
2024-08-09T15:02:22.046060032+03:00 stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-08-09T15:02:22.046062732+03:00 stderr F I0809 12:02:22.046024       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-08-09T15:02:22.814075445+03:00 stderr F I0809 12:02:22.813922       1 trace.go:236] Trace[1025025397]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:15.726) (total time: 7087ms):
2024-08-09T15:02:22.814110545+03:00 stderr F Trace[1025025397]: [7.087572533s] [7.087572533s] END
2024-08-09T15:02:22.814114645+03:00 stderr F I0809 12:02:22.813922       1 trace.go:236] Trace[1335842645]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:15.726) (total time: 7087ms):
2024-08-09T15:02:22.814117445+03:00 stderr F Trace[1335842645]: [7.087771736s] [7.087771736s] END
2024-08-09T15:02:22.814183046+03:00 stderr F I0809 12:02:22.814010       1 trace.go:236] Trace[1225251564]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:15.722) (total time: 7091ms):
2024-08-09T15:02:22.814188547+03:00 stderr F Trace[1225251564]: [7.091668896s] [7.091668896s] END
2024-08-09T15:02:22.814191547+03:00 stderr F I0809 12:02:22.814021       1 trace.go:236] Trace[1562137121]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:15.722) (total time: 7091ms):
2024-08-09T15:02:22.814195347+03:00 stderr F Trace[1562137121]: [7.0918761s] [7.0918761s] END
2024-08-09T15:02:22.814197847+03:00 stderr F I0809 12:02:22.814042       1 trace.go:236] Trace[353690554]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:15.726) (total time: 7087ms):
2024-08-09T15:02:22.814200347+03:00 stderr F Trace[353690554]: [7.087667734s] [7.087667734s] END
2024-08-09T15:02:22.814203047+03:00 stderr F I0809 12:02:22.814082       1 trace.go:236] Trace[162920143]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:15.726) (total time: 7087ms):
2024-08-09T15:02:22.814205947+03:00 stderr F Trace[162920143]: [7.087585133s] [7.087585133s] END
2024-08-09T15:02:22.814375649+03:00 stderr F I0809 12:02:22.814188       1 trace.go:236] Trace[1096454082]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:15.722) (total time: 7091ms):
2024-08-09T15:02:22.81439835+03:00 stderr F Trace[1096454082]: [7.091793498s] [7.091793498s] END
2024-08-09T15:02:22.81440655+03:00 stderr F I0809 12:02:22.814304       1 trace.go:236] Trace[1495827805]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:15.726) (total time: 7088ms):
2024-08-09T15:02:22.81441045+03:00 stderr F Trace[1495827805]: [7.088147442s] [7.088147442s] END
2024-08-09T15:02:22.814590953+03:00 stderr F I0809 12:02:22.814466       1 trace.go:236] Trace[1782644878]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:15.722) (total time: 7092ms):
2024-08-09T15:02:22.814602153+03:00 stderr F Trace[1782644878]: [7.092257706s] [7.092257706s] END
2024-08-09T15:02:22.814658054+03:00 stderr F I0809 12:02:22.814593       1 trace.go:236] Trace[1455662862]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:15.722) (total time: 7092ms):
2024-08-09T15:02:22.814662754+03:00 stderr F Trace[1455662862]: [7.092291206s] [7.092291206s] END
2024-08-09T15:02:23.13421011+03:00 stderr F I0809 12:02:23.134049       1 trace.go:236] Trace[2015540767]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:16.030) (total time: 7103ms):
2024-08-09T15:02:23.134240111+03:00 stderr F Trace[2015540767]: [7.103957787s] [7.103957787s] END
2024-08-09T15:02:24.157898988+03:00 stderr F I0809 12:02:24.157806       1 trace.go:236] Trace[270161365]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:17.030) (total time: 7127ms):
2024-08-09T15:02:24.158123392+03:00 stderr F Trace[270161365]: [7.127120446s] [7.127120446s] END
2024-08-09T15:02:24.158144692+03:00 stderr F E0809 12:02:24.157945       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-08-09T15:02:24.158149492+03:00 stderr F I0809 12:02:24.158036       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-08-09T15:02:25.738767809+03:00 stderr F E0809 12:02:25.738622       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": context deadline exceeded
2024-08-09T15:02:25.746318626+03:00 stderr F E0809 12:02:25.746175       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:27.870044866+03:00 stderr F I0809 12:02:27.869896       1 trace.go:236] Trace[1358482455]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:20.727) (total time: 7142ms):
2024-08-09T15:02:27.870076767+03:00 stderr F Trace[1358482455]: [7.142485984s] [7.142485984s] END
2024-08-09T15:02:27.870080967+03:00 stderr F I0809 12:02:27.869896       1 trace.go:236] Trace[1096433491]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:20.730) (total time: 7138ms):
2024-08-09T15:02:27.870084267+03:00 stderr F Trace[1096433491]: [7.138917229s] [7.138917229s] END
2024-08-09T15:02:27.870087367+03:00 stderr F I0809 12:02:27.869925       1 trace.go:236] Trace[1606108535]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:20.727) (total time: 7142ms):
2024-08-09T15:02:27.870090567+03:00 stderr F Trace[1606108535]: [7.142654487s] [7.142654487s] END
2024-08-09T15:02:27.870102667+03:00 stderr F I0809 12:02:27.870015       1 trace.go:236] Trace[1792873359]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:20.731) (total time: 7138ms):
2024-08-09T15:02:27.870106367+03:00 stderr F Trace[1792873359]: [7.138882428s] [7.138882428s] END
2024-08-09T15:02:27.870109767+03:00 stderr F I0809 12:02:27.870031       1 trace.go:236] Trace[585322278]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:20.730) (total time: 7139ms):
2024-08-09T15:02:27.870112767+03:00 stderr F Trace[585322278]: [7.139105533s] [7.139105533s] END
2024-08-09T15:02:27.870565874+03:00 stderr F I0809 12:02:27.870225       1 trace.go:236] Trace[590276154]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:20.727) (total time: 7142ms):
2024-08-09T15:02:27.870583974+03:00 stderr F Trace[590276154]: [7.14287629s] [7.14287629s] END
2024-08-09T15:02:27.870592475+03:00 stderr F I0809 12:02:27.870357       1 trace.go:236] Trace[2099351004]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:20.731) (total time: 7139ms):
2024-08-09T15:02:27.870596075+03:00 stderr F Trace[2099351004]: [7.139105233s] [7.139105233s] END
2024-08-09T15:02:27.870599475+03:00 stderr F I0809 12:02:27.870367       1 trace.go:236] Trace[921161821]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:20.727) (total time: 7143ms):
2024-08-09T15:02:27.870602575+03:00 stderr F Trace[921161821]: [7.143015493s] [7.143015493s] END
2024-08-09T15:02:27.870627775+03:00 stderr F I0809 12:02:27.870537       1 trace.go:236] Trace[1950048582]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:20.730) (total time: 7139ms):
2024-08-09T15:02:27.870631175+03:00 stderr F Trace[1950048582]: [7.139560839s] [7.139560839s] END
2024-08-09T15:02:27.870634175+03:00 stderr F I0809 12:02:27.870597       1 trace.go:236] Trace[1594670888]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:20.727) (total time: 7143ms):
2024-08-09T15:02:27.870636975+03:00 stderr F Trace[1594670888]: [7.143174695s] [7.143174695s] END
2024-08-09T15:02:29.150010719+03:00 stderr F I0809 12:02:29.149732       1 trace.go:236] Trace[539326390]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:22.046) (total time: 7103ms):
2024-08-09T15:02:29.15003862+03:00 stderr F Trace[539326390]: [7.103460279s] [7.103460279s] END
2024-08-09T15:02:29.15009702+03:00 stderr F E0809 12:02:29.149936       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.3.2:10250, code 502: 502 Bad Gateway
2024-08-09T15:02:29.150108621+03:00 stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-08-09T15:02:30.739598275+03:00 stderr F E0809 12:02:30.739521       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": context deadline exceeded
2024-08-09T15:02:30.752722578+03:00 stderr F E0809 12:02:30.751159       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:32.862675705+03:00 stderr F I0809 12:02:32.862397       1 trace.go:236] Trace[1965973085]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:25.746) (total time: 7115ms):
2024-08-09T15:02:32.862701405+03:00 stderr F Trace[1965973085]: [7.115979873s] [7.115979873s] END
2024-08-09T15:02:32.862706605+03:00 stderr F I0809 12:02:32.862420       1 trace.go:236] Trace[2058601602]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:25.746) (total time: 7115ms):
2024-08-09T15:02:32.862709505+03:00 stderr F Trace[2058601602]: [7.115963873s] [7.115963873s] END
2024-08-09T15:02:32.862728706+03:00 stderr F I0809 12:02:32.862553       1 trace.go:236] Trace[1767804245]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:25.738) (total time: 7123ms):
2024-08-09T15:02:32.862732006+03:00 stderr F Trace[1767804245]: [7.123553091s] [7.123553091s] END
2024-08-09T15:02:32.862815107+03:00 stderr F I0809 12:02:32.862671       1 trace.go:236] Trace[1744066289]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:25.746) (total time: 7116ms):
2024-08-09T15:02:32.862821407+03:00 stderr F Trace[1744066289]: [7.116243178s] [7.116243178s] END
2024-08-09T15:02:32.862824707+03:00 stderr F I0809 12:02:32.862701       1 trace.go:236] Trace[830441511]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:25.746) (total time: 7116ms):
2024-08-09T15:02:32.862827507+03:00 stderr F Trace[830441511]: [7.11639058s] [7.11639058s] END
2024-08-09T15:02:32.862911709+03:00 stderr F I0809 12:02:32.862823       1 trace.go:236] Trace[1963766700]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:25.738) (total time: 7124ms):
2024-08-09T15:02:32.862924609+03:00 stderr F Trace[1963766700]: [7.124045199s] [7.124045199s] END
2024-08-09T15:02:32.863343515+03:00 stderr F I0809 12:02:32.863034       1 trace.go:236] Trace[1259550035]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:25.738) (total time: 7124ms):
2024-08-09T15:02:32.863360416+03:00 stderr F Trace[1259550035]: [7.1241298s] [7.1241298s] END
2024-08-09T15:02:32.863364116+03:00 stderr F I0809 12:02:32.863069       1 trace.go:236] Trace[82246690]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:25.738) (total time: 7124ms):
2024-08-09T15:02:32.863367616+03:00 stderr F Trace[82246690]: [7.124259302s] [7.124259302s] END
2024-08-09T15:02:32.863370516+03:00 stderr F I0809 12:02:32.863272       1 trace.go:236] Trace[712678778]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:25.738) (total time: 7124ms):
2024-08-09T15:02:32.863373516+03:00 stderr F Trace[712678778]: [7.124369604s] [7.124369604s] END
2024-08-09T15:02:32.863415316+03:00 stderr F I0809 12:02:32.863342       1 trace.go:236] Trace[1886041973]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:25.746) (total time: 7116ms):
2024-08-09T15:02:32.863421617+03:00 stderr F Trace[1886041973]: [7.116994689s] [7.116994689s] END
2024-08-09T15:02:35.749123476+03:00 stderr F E0809 12:02:35.748967       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1: Get "https://10.42.1.4:6443/apis/custom.metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:35.759200232+03:00 stderr F E0809 12:02:35.759079       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.42.3.2:10250/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2024-08-09T15:02:35.774018762+03:00 stderr F E0809 12:02:35.773930       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:35.804904341+03:00 stderr F E0809 12:02:35.804776       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-08-09T15:02:36.253981906+03:00 stderr F I0809 12:02:36.253850       1 trace.go:236] Trace[1453920203]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:29.150) (total time: 7103ms):
2024-08-09T15:02:36.254014107+03:00 stderr F Trace[1453920203]: [7.103541981s] [7.103541981s] END
2024-08-09T15:02:36.254026807+03:00 stderr F E0809 12:02:36.253965       1 controller.go:146] Error updating APIService "v1beta1.custom.metrics.k8s.io" with err: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: proxy error from 127.0.0.1:9345 while dialing 10.42.1.4:6443, code 502: 502 Bad Gateway
2024-08-09T15:02:36.254030307+03:00 stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-08-09T15:02:36.254090508+03:00 stderr F I0809 12:02:36.254017       1 trace.go:236] Trace[1494314013]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:29.150) (total time: 7103ms):
2024-08-09T15:02:36.254094308+03:00 stderr F Trace[1494314013]: [7.103739083s] [7.103739083s] END
2024-08-09T15:02:36.273112803+03:00 stderr F I0809 12:02:36.272989       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:36.348568873+03:00 stderr F I0809 12:02:36.348404       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:02:37.279027705+03:00 stderr F I0809 12:02:37.278877       1 trace.go:236] Trace[669851300]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.150) (total time: 7128ms):
2024-08-09T15:02:37.279069706+03:00 stderr F Trace[669851300]: [7.128220564s] [7.128220564s] END
2024-08-09T15:02:37.855408845+03:00 stderr F I0809 12:02:37.855066       1 trace.go:236] Trace[374571786]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.751) (total time: 7103ms):
2024-08-09T15:02:37.855459446+03:00 stderr F Trace[374571786]: [7.103744184s] [7.103744184s] END
2024-08-09T15:02:37.855465546+03:00 stderr F I0809 12:02:37.855066       1 trace.go:236] Trace[594166538]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.751) (total time: 7103ms):
2024-08-09T15:02:37.855472446+03:00 stderr F Trace[594166538]: [7.103174475s] [7.103174475s] END
2024-08-09T15:02:37.855475246+03:00 stderr F I0809 12:02:37.855106       1 trace.go:236] Trace[488131089]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:30.741) (total time: 7113ms):
2024-08-09T15:02:37.855478647+03:00 stderr F Trace[488131089]: [7.113913341s] [7.113913341s] END
2024-08-09T15:02:37.855481147+03:00 stderr F I0809 12:02:37.855304       1 trace.go:236] Trace[2051829130]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.751) (total time: 7103ms):
2024-08-09T15:02:37.855483447+03:00 stderr F Trace[2051829130]: [7.103682482s] [7.103682482s] END
2024-08-09T15:02:37.855486247+03:00 stderr F I0809 12:02:37.855311       1 trace.go:236] Trace[31621103]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.751) (total time: 7103ms):
2024-08-09T15:02:37.855494447+03:00 stderr F Trace[31621103]: [7.103541481s] [7.103541481s] END
2024-08-09T15:02:37.855497947+03:00 stderr F I0809 12:02:37.855339       1 trace.go:236] Trace[48358201]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:30.739) (total time: 7115ms):
2024-08-09T15:02:37.855501047+03:00 stderr F Trace[48358201]: [7.115666468s] [7.115666468s] END
2024-08-09T15:02:37.855517447+03:00 stderr F I0809 12:02:37.855389       1 trace.go:236] Trace[1104828605]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:30.741) (total time: 7113ms):
2024-08-09T15:02:37.855537647+03:00 stderr F Trace[1104828605]: [7.11383684s] [7.11383684s] END
2024-08-09T15:02:37.855542148+03:00 stderr F I0809 12:02:37.855419       1 trace.go:236] Trace[1683252658]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:30.741) (total time: 7113ms):
2024-08-09T15:02:37.855545548+03:00 stderr F Trace[1683252658]: [7.113717939s] [7.113717939s] END
2024-08-09T15:02:37.855557548+03:00 stderr F I0809 12:02:37.855511       1 trace.go:236] Trace[1885161327]: "Proxy via http_connect protocol over tcp" address:10.42.3.2:10250 (09-Aug-2024 12:02:30.751) (total time: 7103ms):
2024-08-09T15:02:37.855561848+03:00 stderr F Trace[1885161327]: [7.103547881s] [7.103547881s] END
2024-08-09T15:02:37.85568175+03:00 stderr F I0809 12:02:37.855608       1 trace.go:236] Trace[1422420893]: "Proxy via http_connect protocol over tcp" address:10.42.1.4:6443 (09-Aug-2024 12:02:30.740) (total time: 7114ms):
2024-08-09T15:02:37.85569005+03:00 stderr F Trace[1422420893]: [7.114609452s] [7.114609452s] END
2024-08-09T15:04:05.799070808+03:00 stderr F I0809 12:04:05.798942       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:05:05.799670855+03:00 stderr F I0809 12:05:05.799522       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:07:00.717429848+03:00 stderr F I0809 12:07:00.717282       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:09:05.800124268+03:00 stderr F I0809 12:09:05.799980       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:10:05.790328511+03:00 stderr F I0809 12:10:05.790111       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:12:00.730786226+03:00 stderr F I0809 12:12:00.730543       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:12:00.731327735+03:00 stderr F E0809 12:12:00.731212       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-08-09T15:12:00.731463037+03:00 stderr F I0809 12:12:00.731388       1 trace.go:236] Trace[1344347633]: "Get" accept:,audit-id:41d9c7e8-3512-4a98-a031-fba57fe5d53a,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (09-Aug-2024 12:07:57.073) (total time: 243658ms):
2024-08-09T15:12:00.731481537+03:00 stderr F Trace[1344347633]: ---"Writing http response done" 243654ms (12:12:00.731)
2024-08-09T15:12:00.731491037+03:00 stderr F Trace[1344347633]: [4m3.658226816s] [4m3.658226816s] END
2024-08-09T15:14:05.799036423+03:00 stderr F I0809 12:14:05.797658       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:15:05.80178886+03:00 stderr F I0809 12:15:05.801648       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:17:00.73221562+03:00 stderr F I0809 12:17:00.731881       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:19:05.798208782+03:00 stderr F I0809 12:19:05.798095       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:20:05.79522873+03:00 stderr F I0809 12:20:05.795103       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:22:00.749767764+03:00 stderr F I0809 12:22:00.749633       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:23:05.804569456+03:00 stderr F I0809 12:23:05.804433       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:25:05.804217623+03:00 stderr F I0809 12:25:05.804089       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:27:00.746321649+03:00 stderr F I0809 12:27:00.746109       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:28:05.794907037+03:00 stderr F I0809 12:28:05.794780       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:30:05.796403133+03:00 stderr F I0809 12:30:05.796263       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:32:00.768813828+03:00 stderr F I0809 12:32:00.768688       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-08-09T15:33:05.794794166+03:00 stderr F I0809 12:33:05.794658       1 handler.go:286] Adding GroupVersion custom

cengizhanR Aug 14, 2024
Author

This is etcd logs, @brandond


2024-08-09T15:01:25.150650048+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.15015Z","caller":"etcdmain/config.go:350","msg":"loaded server configuration, other configuration command line flags and environment variables will be ignored if provided","path":"/var/lib/rancher/rke2/server/db/etcd/config"}
2024-08-09T15:01:25.150682349+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.150541Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--config-file=/var/lib/rancher/rke2/server/db/etcd/config"]}
2024-08-09T15:01:25.15074975+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:25.15068Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"config","data-dir":"/var/lib/rancher/rke2/server/db/etcd"}
2024-08-09T15:01:25.15076025+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:25.150699Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"name","data-dir":"/var/lib/rancher/rke2/server/db/etcd"}
2024-08-09T15:01:25.15076215+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.150704Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/rancher/rke2/server/db/etcd","dir-type":"member"}
2024-08-09T15:01:25.15078685+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.150724Z","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://10.0.0.2:2380","https://127.0.0.1:2380"]}
2024-08-09T15:01:25.150804351+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.150763Z","caller":"embed/etcd.go:495","msg":"starting with peer TLS","tls-info":"cert = /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
2024-08-09T15:01:25.151339259+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.151269Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://10.0.0.2:2379","https://127.0.0.1:2379"]}
2024-08-09T15:01:25.151457361+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.151378Z","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.20.7 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"rke21-a0cbe7f5","data-dir":"/var/lib/rancher/rke2/server/db/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["https://10.0.0.2:2380","https://127.0.0.1:2380"],"advertise-client-urls":["https://10.0.0.2:2379"],"listen-client-urls":["https://10.0.0.2:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://10.0.0.2:2381","http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
2024-08-09T15:01:25.161852522+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:25.161746Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","took":"10.012055ms"}
2024-08-09T15:01:25.162317529+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:25.162095Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"000000000000003e-00000000004a0c4f.wal.broken"}
2024-08-09T15:01:26.011666303+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.011363Z","caller":"etcdserver/server.go:509","msg":"recovered v2 store from snapshot","snapshot-index":28205985,"snapshot-size":"22 kB"}
2024-08-09T15:01:26.011689803+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.011555Z","caller":"etcdserver/server.go:522","msg":"recovered v3 backend from snapshot","backend-size-bytes":48406528,"backend-size":"48 MB","backend-size-in-use-bytes":28487680,"backend-size-in-use":"28 MB"}
2024-08-09T15:01:26.011694704+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.01161Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"000000000000003e-00000000004a0c4f.wal.broken"}
2024-08-09T15:01:26.176227556+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.176077Z","caller":"etcdserver/raft.go:530","msg":"restarting local member","cluster-id":"6300d4a278043cb","local-member-id":"86e02918fafb02ec","commit-index":28210252}
2024-08-09T15:01:26.176816065+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.176722Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"86e02918fafb02ec switched to configuration voters=(5947303585021372337 9718813183132238572 17155969106340062836)"}
2024-08-09T15:01:26.176827665+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.176773Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"86e02918fafb02ec became follower at term 15"}
2024-08-09T15:01:26.176863065+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.176794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 86e02918fafb02ec [peers: [52891119b49b27b1,86e02918fafb02ec,ee163f0188f9a674], term: 15, commit: 28210252, applied: 28205985, lastindex: 28210253, lastterm: 15]"}
2024-08-09T15:01:26.17714027+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.176957Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
2024-08-09T15:01:26.17715137+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.177075Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6300d4a278043cb","local-member-id":"86e02918fafb02ec","recovered-remote-peer-id":"52891119b49b27b1","recovered-remote-peer-urls":["https://10.0.0.3:2380"]}
2024-08-09T15:01:26.17715427+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.177094Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6300d4a278043cb","local-member-id":"86e02918fafb02ec","recovered-remote-peer-id":"86e02918fafb02ec","recovered-remote-peer-urls":["https://10.0.0.2:2380"]}
2024-08-09T15:01:26.17715657+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.177103Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6300d4a278043cb","local-member-id":"86e02918fafb02ec","recovered-remote-peer-id":"ee163f0188f9a674","recovered-remote-peer-urls":["https://10.0.0.4:2380"]}
2024-08-09T15:01:26.17717417+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.177114Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
2024-08-09T15:01:26.178554792+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.178472Z","caller":"auth/store.go:1238","msg":"simple token is not cryptographically signed"}
2024-08-09T15:01:26.179380004+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.179294Z","caller":"mvcc/kvstore.go:323","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":47802872}
2024-08-09T15:01:26.203948186+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.203806Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":47807281}
2024-08-09T15:01:26.207568942+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.20748Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
2024-08-09T15:01:26.209735775+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.209626Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.209751876+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.20967Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.210075081+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.209983Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.210309884+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210208Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.210594789+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210351Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.210616589+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210394Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1","remote-peer-urls":["https://10.0.0.3:2380"]}
2024-08-09T15:01:26.210627989+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210422Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.21070909+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210634Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.210800892+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210405Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.210989795+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210396Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.211001995+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.210916Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.211140997+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211022Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.211237199+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211154Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.211244599+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211194Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674","remote-peer-urls":["https://10.0.0.4:2380"]}
2024-08-09T15:01:26.211517903+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211205Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.211743106+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211327Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.211761007+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.211691Z","caller":"etcdserver/corrupt.go:95","msg":"starting initial corruption check","local-member-id":"86e02918fafb02ec","timeout":"15s"}
2024-08-09T15:01:26.2177811+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.217667Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.2177985+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.217694Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.218523712+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.218418Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.218986219+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.218914Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.218995019+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.218936Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.221400556+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.221313Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.225271116+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.225173Z","caller":"etcdserver/corrupt.go:398","msg":"failed hash kv request","local-member-id":"86e02918fafb02ec","requested-revision":47807281,"remote-peer-endpoint":"https://10.0.0.3:2380","error":"etcdserver: mvcc: required revision has been compacted"}
2024-08-09T15:01:26.229562683+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.229464Z","caller":"etcdserver/corrupt.go:398","msg":"failed hash kv request","local-member-id":"86e02918fafb02ec","requested-revision":47807281,"remote-peer-endpoint":"https://10.0.0.4:2380","error":"etcdserver: mvcc: required revision has been compacted"}
2024-08-09T15:01:26.229581183+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.22949Z","caller":"etcdserver/corrupt.go:165","msg":"initial corruption checking passed; no corruption","local-member-id":"86e02918fafb02ec"}
2024-08-09T15:01:26.229630284+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.229518Z","caller":"etcdserver/server.go:845","msg":"starting etcd server","local-member-id":"86e02918fafb02ec","local-server-version":"3.5.9","cluster-id":"6300d4a278043cb","cluster-version":"3.5"}
2024-08-09T15:01:26.229699985+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.229611Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/rancher/rke2/server/db/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
2024-08-09T15:01:26.229705385+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.229639Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
2024-08-09T15:01:26.229707885+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.229658Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/rancher/rke2/server/db/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
2024-08-09T15:01:26.229713385+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.229674Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/rancher/rke2/server/db/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
2024-08-09T15:01:26.231821318+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231719Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /var/lib/rancher/rke2/server/tls/etcd/server-client.crt, key = /var/lib/rancher/rke2/server/tls/etcd/server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
2024-08-09T15:01:26.231895619+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231766Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"10.0.0.2:2380"}
2024-08-09T15:01:26.231903819+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231774Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"127.0.0.1:2380"}
2024-08-09T15:01:26.231907519+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231788Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"10.0.0.2:2380"}
2024-08-09T15:01:26.231909719+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231795Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"127.0.0.1:2380"}
2024-08-09T15:01:26.232004321+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231916Z","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"86e02918fafb02ec","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["https://10.0.0.2:2380","https://127.0.0.1:2380"],"advertise-client-urls":["https://10.0.0.2:2379"],"listen-client-urls":["https://10.0.0.2:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://10.0.0.2:2381","http://127.0.0.1:2381"]}
2024-08-09T15:01:26.232013121+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.231952Z","caller":"embed/etcd.go:855","msg":"serving metrics","address":"http://127.0.0.1:2381"}
2024-08-09T15:01:26.232231724+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232167Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"86e02918fafb02ec","to":"ee163f0188f9a674","stream-type":"stream Message"}
2024-08-09T15:01:26.232236624+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232197Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.232360826+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232301Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 86e02918fafb02ec elected leader ee163f0188f9a674 at term 15"}
2024-08-09T15:01:26.232400727+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.23235Z","caller":"embed/etcd.go:855","msg":"serving metrics","address":"http://10.0.0.2:2381"}
2024-08-09T15:01:26.232555629+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232459Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"86e02918fafb02ec","to":"52891119b49b27b1","stream-type":"stream Message"}
2024-08-09T15:01:26.232559329+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232501Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.232677331+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232546Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"86e02918fafb02ec","to":"ee163f0188f9a674","stream-type":"stream MsgApp v2"}
2024-08-09T15:01:26.232799733+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232571Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"ee163f0188f9a674"}
2024-08-09T15:01:26.233067637+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232957Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"86e02918fafb02ec","to":"52891119b49b27b1","stream-type":"stream MsgApp v2"}
2024-08-09T15:01:26.233081637+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.232998Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"86e02918fafb02ec","remote-peer-id":"52891119b49b27b1"}
2024-08-09T15:01:26.282388202+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.280495Z","caller":"etcdserver/server.go:777","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"86e02918fafb02ec","forward-ticks":8,"forward-duration":"4s","election-ticks":10,"election-timeout":"5s","active-remote-members":2}
2024-08-09T15:01:26.290864434+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.290665Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47805533}
2024-08-09T15:01:26.376935569+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.376783Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47805533,"took":"84.439909ms","hash":3220121351}
2024-08-09T15:01:26.376955369+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.376841Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3220121351,"revision":47805533,"compact-revision":47802872}
2024-08-09T15:01:26.376958469+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.376856Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47807816}
2024-08-09T15:01:26.426456737+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426317Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
2024-08-09T15:01:26.426482037+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426357Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
2024-08-09T15:01:26.426486737+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426327Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
2024-08-09T15:01:26.426524138+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426459Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
2024-08-09T15:01:26.426549138+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426499Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
2024-08-09T15:01:26.42666564+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.426335Z","caller":"etcdserver/server.go:2064","msg":"published local member to cluster through raft","local-member-id":"86e02918fafb02ec","local-member-attributes":"{Name:rke21-a0cbe7f5 ClientURLs:[https://10.0.0.2:2379]}","request-path":"/0/members/86e02918fafb02ec/attributes","cluster-id":"6300d4a278043cb","publish-timeout":"15s"}
2024-08-09T15:01:26.427747557+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.42763Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"http","address":"127.0.0.1:2382"}
2024-08-09T15:01:26.427778057+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.427691Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc","address":"10.0.0.2:2379"}
2024-08-09T15:01:26.428191664+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.427701Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc","address":"127.0.0.1:2379"}
2024-08-09T15:01:26.439768543+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.439674Z","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
2024-08-09T15:01:26.443031994+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.442947Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","current-db-size-bytes":48406528,"current-db-size":"48 MB","current-db-size-in-use-bytes":29224960,"current-db-size-in-use":"29 MB"}
2024-08-09T15:01:26.864201426+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.863994Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","current-db-size-bytes-diff":-19955712,"current-db-size-bytes":28450816,"current-db-size":"28 MB","current-db-size-in-use-bytes-diff":-782336,"current-db-size-in-use-bytes":28442624,"current-db-size-in-use":"28 MB","took":"424.273481ms"}
2024-08-09T15:01:26.864233027+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.864071Z","caller":"v3rpc/maintenance.go:96","msg":"finished defragment"}
2024-08-09T15:01:26.864237927+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.864095Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-08-09T12:01:26.439659Z","time spent":"424.431383ms","remote":"127.0.0.1:48436","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-08-09T15:01:26.865036839+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:26.864927Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"399.643098ms","expected-duration":"100ms","prefix":"","request":"header:<ID:2860226881285433831 username:\"etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/cattle-provisioning-capi-system/controller-leader-election-capi\" mod_revision:47810998 > success:<request_put:<key:\"/registry/leases/cattle-provisioning-capi-system/controller-leader-election-capi\" value_size:474 >> failure:<>>","response":"size:20"}
2024-08-09T15:01:26.86573475+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.865619Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47807816,"took":"487.204756ms","hash":1927520194}
2024-08-09T15:01:26.865753451+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:26.865654Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1927520194,"revision":47807816,"compact-revision":47805533}
2024-08-09T15:01:58.551869322+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:58.551755Z","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
2024-08-09T15:01:58.555220374+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:58.555119Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","current-db-size-bytes":28610560,"current-db-size":"29 MB","current-db-size-in-use-bytes":28262400,"current-db-size-in-use":"28 MB"}
2024-08-09T15:01:58.817003135+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:58.816823Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","current-db-size-bytes-diff":-389120,"current-db-size-bytes":28221440,"current-db-size":"28 MB","current-db-size-in-use-bytes-diff":-49152,"current-db-size-in-use-bytes":28213248,"current-db-size-in-use":"28 MB","took":"265.026511ms"}
2024-08-09T15:01:58.817026435+03:00 stderr F {"level":"info","ts":"2024-08-09T12:01:58.816896Z","caller":"v3rpc/maintenance.go:96","msg":"finished defragment"}
2024-08-09T15:01:58.81735204+03:00 stderr F {"level":"warn","ts":"2024-08-09T12:01:58.817225Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"184.526262ms","expected-duration":"100ms","prefix":"","request":"header:<ID:2860226881285434233 username:\"etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/cattle-provisioning-capi-system/controller-leader-election-capi\" mod_revision:47811231 > success:<request_put:<key:\"/registry/leases/cattle-provisioning-capi-system/controller-leader-election-capi\" value_size:474 >> failure:<>>","response":"size:20"}
2024-08-09T15:04:11.35671511+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:11.356536Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47809898}
2024-08-09T15:04:11.423346144+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:11.42319Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47809898,"took":"65.550916ms","hash":1743299878}
2024-08-09T15:04:11.423377544+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:11.423239Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1743299878,"revision":47809898,"compact-revision":47807816}
2024-08-09T15:04:21.70958729+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:21.709435Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28215986,"local-member-snapshot-index":28205985,"local-member-snapshot-count":10000}
2024-08-09T15:04:21.715443481+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:21.715298Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28215986}
2024-08-09T15:04:21.715483482+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:21.715439Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28210986}
2024-08-09T15:04:26.234474974+03:00 stderr F {"level":"info","ts":"2024-08-09T12:04:26.234283Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001adc75c.snap"}
2024-08-09T15:09:11.366223998+03:00 stderr F {"level":"info","ts":"2024-08-09T12:09:11.366043Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47812604}
2024-08-09T15:09:11.447678561+03:00 stderr F {"level":"info","ts":"2024-08-09T12:09:11.447502Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47812604,"took":"80.352547ms","hash":855335572}
2024-08-09T15:09:11.447712862+03:00 stderr F {"level":"info","ts":"2024-08-09T12:09:11.447552Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":855335572,"revision":47812604,"compact-revision":47809898}
2024-08-09T15:14:11.371571552+03:00 stderr F {"level":"info","ts":"2024-08-09T12:14:11.371361Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47814880}
2024-08-09T15:14:11.434944435+03:00 stderr F {"level":"info","ts":"2024-08-09T12:14:11.434772Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47814880,"took":"62.369468ms","hash":3190796232}
2024-08-09T15:14:11.434980136+03:00 stderr F {"level":"info","ts":"2024-08-09T12:14:11.434827Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3190796232,"revision":47814880,"compact-revision":47812604}
2024-08-09T15:19:11.379963754+03:00 stderr F {"level":"info","ts":"2024-08-09T12:19:11.379602Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47817157}
2024-08-09T15:19:11.442124818+03:00 stderr F {"level":"info","ts":"2024-08-09T12:19:11.44197Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47817157,"took":"61.413153ms","hash":2130426632}
2024-08-09T15:19:11.442167619+03:00 stderr F {"level":"info","ts":"2024-08-09T12:19:11.442019Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2130426632,"revision":47817157,"compact-revision":47814880}
2024-08-09T15:24:11.384696189+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:11.384515Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47819431}
2024-08-09T15:24:11.446159142+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:11.445978Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47819431,"took":"60.286735ms","hash":1155704107}
2024-08-09T15:24:11.446197043+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:11.446034Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1155704107,"revision":47819431,"compact-revision":47817157}
2024-08-09T15:24:42.211430604+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:42.211221Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28225987,"local-member-snapshot-index":28215986,"local-member-snapshot-count":10000}
2024-08-09T15:24:42.216820988+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:42.216692Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28225987}
2024-08-09T15:24:42.21693699+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:42.216867Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28220987}
2024-08-09T15:24:56.267767215+03:00 stderr F {"level":"info","ts":"2024-08-09T12:24:56.267571Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001adee6d.snap"}
2024-08-09T15:29:11.392373039+03:00 stderr F {"level":"info","ts":"2024-08-09T12:29:11.39216Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47821710}
2024-08-09T15:29:11.454912609+03:00 stderr F {"level":"info","ts":"2024-08-09T12:29:11.454747Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47821710,"took":"61.454753ms","hash":692312393}
2024-08-09T15:29:11.454957509+03:00 stderr F {"level":"info","ts":"2024-08-09T12:29:11.454799Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":692312393,"revision":47821710,"compact-revision":47819431}
2024-08-09T15:34:11.397414248+03:00 stderr F {"level":"info","ts":"2024-08-09T12:34:11.397216Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47823981}
2024-08-09T15:34:11.457595781+03:00 stderr F {"level":"info","ts":"2024-08-09T12:34:11.45743Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47823981,"took":"59.26782ms","hash":2521504181}
2024-08-09T15:34:11.457637082+03:00 stderr F {"level":"info","ts":"2024-08-09T12:34:11.457483Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2521504181,"revision":47823981,"compact-revision":47821710}
2024-08-09T15:39:11.404115683+03:00 stderr F {"level":"info","ts":"2024-08-09T12:39:11.403931Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47826253}
2024-08-09T15:39:11.462651191+03:00 stderr F {"level":"info","ts":"2024-08-09T12:39:11.462507Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47826253,"took":"57.560792ms","hash":3178569157}
2024-08-09T15:39:11.462677991+03:00 stderr F {"level":"info","ts":"2024-08-09T12:39:11.462554Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3178569157,"revision":47826253,"compact-revision":47823981}
2024-08-09T15:39:32.359881502+03:00 stderr F {"level":"info","ts":"2024-08-09T12:39:32.359738Z","caller":"wal/wal.go:785","msg":"created a new WAL segment","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000000167-0000000001aece43.wal"}
2024-08-09T15:39:56.293223302+03:00 stderr F {"level":"info","ts":"2024-08-09T12:39:56.293024Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000000162-0000000001a83e19.wal"}
2024-08-09T15:44:11.410852657+03:00 stderr F {"level":"info","ts":"2024-08-09T12:44:11.410654Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47828537}
2024-08-09T15:44:11.470677285+03:00 stderr F {"level":"info","ts":"2024-08-09T12:44:11.470497Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47828537,"took":"58.774511ms","hash":927873895}
2024-08-09T15:44:11.470721986+03:00 stderr F {"level":"info","ts":"2024-08-09T12:44:11.470552Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":927873895,"revision":47828537,"compact-revision":47826253}
2024-08-09T15:45:02.940447663+03:00 stderr F {"level":"info","ts":"2024-08-09T12:45:02.940234Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28235988,"local-member-snapshot-index":28225987,"local-member-snapshot-count":10000}
2024-08-09T15:45:02.944810731+03:00 stderr F {"level":"info","ts":"2024-08-09T12:45:02.944697Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28235988}
2024-08-09T15:45:02.944983633+03:00 stderr F {"level":"info","ts":"2024-08-09T12:45:02.94491Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28230988}
2024-08-09T15:45:26.292650693+03:00 stderr F {"level":"info","ts":"2024-08-09T12:45:26.292506Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001ae157e.snap"}
2024-08-09T15:49:11.42098849+03:00 stderr F {"level":"info","ts":"2024-08-09T12:49:11.420801Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47830805}
2024-08-09T15:49:11.48547519+03:00 stderr F {"level":"info","ts":"2024-08-09T12:49:11.485154Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47830805,"took":"63.21808ms","hash":541222980}
2024-08-09T15:49:11.485515491+03:00 stderr F {"level":"info","ts":"2024-08-09T12:49:11.485201Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":541222980,"revision":47830805,"compact-revision":47828537}
2024-08-09T15:54:11.423914911+03:00 stderr F {"level":"info","ts":"2024-08-09T12:54:11.423742Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47833081}
2024-08-09T15:54:11.483521435+03:00 stderr F {"level":"info","ts":"2024-08-09T12:54:11.483373Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47833081,"took":"58.638309ms","hash":286105030}
2024-08-09T15:54:11.483559636+03:00 stderr F {"level":"info","ts":"2024-08-09T12:54:11.483417Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":286105030,"revision":47833081,"compact-revision":47830805}
2024-08-09T15:59:11.430994918+03:00 stderr F {"level":"info","ts":"2024-08-09T12:59:11.430803Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47835360}
2024-08-09T15:59:11.502767431+03:00 stderr F {"level":"info","ts":"2024-08-09T12:59:11.502591Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47835360,"took":"66.607633ms","hash":1851878725}
2024-08-09T15:59:11.502805132+03:00 stderr F {"level":"info","ts":"2024-08-09T12:59:11.502636Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1851878725,"revision":47835360,"compact-revision":47833081}
2024-08-09T16:04:11.444138784+03:00 stderr F {"level":"info","ts":"2024-08-09T13:04:11.443958Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47837633}
2024-08-09T16:04:11.50512543+03:00 stderr F {"level":"info","ts":"2024-08-09T13:04:11.504898Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47837633,"took":"60.079032ms","hash":467944257}
2024-08-09T16:04:11.505170531+03:00 stderr F {"level":"info","ts":"2024-08-09T13:04:11.504949Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":467944257,"revision":47837633,"compact-revision":47835360}
2024-08-09T16:05:21.435044634+03:00 stderr F {"level":"info","ts":"2024-08-09T13:05:21.43482Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28245989,"local-member-snapshot-index":28235988,"local-member-snapshot-count":10000}
2024-08-09T16:05:21.439554304+03:00 stderr F {"level":"info","ts":"2024-08-09T13:05:21.43935Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28245989}
2024-08-09T16:05:21.439616905+03:00 stderr F {"level":"info","ts":"2024-08-09T13:05:21.439555Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28240989}
2024-08-09T16:05:26.320836312+03:00 stderr F {"level":"info","ts":"2024-08-09T13:05:26.320719Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001ae3c90.snap"}
2024-08-09T16:09:11.452692879+03:00 stderr F {"level":"info","ts":"2024-08-09T13:09:11.452323Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47839915}
2024-08-09T16:09:11.517783689+03:00 stderr F {"level":"info","ts":"2024-08-09T13:09:11.51762Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47839915,"took":"63.674887ms","hash":447992950}
2024-08-09T16:09:11.51783169+03:00 stderr F {"level":"info","ts":"2024-08-09T13:09:11.517671Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":447992950,"revision":47839915,"compact-revision":47837633}
2024-08-09T16:14:11.460639665+03:00 stderr F {"level":"info","ts":"2024-08-09T13:14:11.460461Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47842192}
2024-08-09T16:14:11.522593626+03:00 stderr F {"level":"info","ts":"2024-08-09T13:14:11.522439Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47842192,"took":"61.015546ms","hash":3679574448}
2024-08-09T16:14:11.522633126+03:00 stderr F {"level":"info","ts":"2024-08-09T13:14:11.522477Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3679574448,"revision":47842192,"compact-revision":47839915}
2024-08-09T16:19:11.468905936+03:00 stderr F {"level":"info","ts":"2024-08-09T13:19:11.468707Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47844470}
2024-08-09T16:19:11.53233812+03:00 stderr F {"level":"info","ts":"2024-08-09T13:19:11.532154Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47844470,"took":"62.50647ms","hash":3633002351}
2024-08-09T16:19:11.532385621+03:00 stderr F {"level":"info","ts":"2024-08-09T13:19:11.532203Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3633002351,"revision":47844470,"compact-revision":47842192}
2024-08-09T16:24:11.475495141+03:00 stderr F {"level":"info","ts":"2024-08-09T13:24:11.475336Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47846746}
2024-08-09T16:24:11.535763975+03:00 stderr F {"level":"info","ts":"2024-08-09T13:24:11.535627Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47846746,"took":"59.172418ms","hash":1321321829}
2024-08-09T16:24:11.535787776+03:00 stderr F {"level":"info","ts":"2024-08-09T13:24:11.535676Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1321321829,"revision":47846746,"compact-revision":47844470}
2024-08-09T16:25:42.2689855+03:00 stderr F {"level":"info","ts":"2024-08-09T13:25:42.268821Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28255990,"local-member-snapshot-index":28245989,"local-member-snapshot-count":10000}
2024-08-09T16:25:42.273777774+03:00 stderr F {"level":"info","ts":"2024-08-09T13:25:42.273495Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28255990}
2024-08-09T16:25:42.273880276+03:00 stderr F {"level":"info","ts":"2024-08-09T13:25:42.273779Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28250990}
2024-08-09T16:25:56.35196765+03:00 stderr F {"level":"info","ts":"2024-08-09T13:25:56.351808Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001ae63a1.snap"}
2024-08-09T16:29:11.482726655+03:00 stderr F {"level":"info","ts":"2024-08-09T13:29:11.482492Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47849017}
2024-08-09T16:29:11.547954067+03:00 stderr F {"level":"info","ts":"2024-08-09T13:29:11.545191Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47849017,"took":"61.754758ms","hash":1394128950}
2024-08-09T16:29:11.547994168+03:00 stderr F {"level":"info","ts":"2024-08-09T13:29:11.545276Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1394128950,"revision":47849017,"compact-revision":47846746}
2024-08-09T16:34:11.488971688+03:00 stderr F {"level":"info","ts":"2024-08-09T13:34:11.488803Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47851296}
2024-08-09T16:34:11.550194238+03:00 stderr F {"level":"info","ts":"2024-08-09T13:34:11.550034Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47851296,"took":"60.333136ms","hash":1181563610}
2024-08-09T16:34:11.550227539+03:00 stderr F {"level":"info","ts":"2024-08-09T13:34:11.550081Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1181563610,"revision":47851296,"compact-revision":47849017}
2024-08-09T16:39:11.49552929+03:00 stderr F {"level":"info","ts":"2024-08-09T13:39:11.495359Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47853569}
2024-08-09T16:39:11.55807686+03:00 stderr F {"level":"info","ts":"2024-08-09T13:39:11.557919Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47853569,"took":"61.630856ms","hash":3939808706}
2024-08-09T16:39:11.558112161+03:00 stderr F {"level":"info","ts":"2024-08-09T13:39:11.557967Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3939808706,"revision":47853569,"compact-revision":47851296}
2024-08-09T16:44:11.50203229+03:00 stderr F {"level":"info","ts":"2024-08-09T13:44:11.501642Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47855848}
2024-08-09T16:44:11.56256783+03:00 stderr F {"level":"info","ts":"2024-08-09T13:44:11.562423Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47855848,"took":"59.726826ms","hash":565534223}
2024-08-09T16:44:11.56260183+03:00 stderr F {"level":"info","ts":"2024-08-09T13:44:11.562468Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":565534223,"revision":47855848,"compact-revision":47853569}
2024-08-09T16:46:02.7868508+03:00 stderr F {"level":"info","ts":"2024-08-09T13:46:02.786665Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28265991,"local-member-snapshot-index":28255990,"local-member-snapshot-count":10000}
2024-08-09T16:46:02.793830408+03:00 stderr F {"level":"info","ts":"2024-08-09T13:46:02.793699Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28265991}
2024-08-09T16:46:02.794064612+03:00 stderr F {"level":"info","ts":"2024-08-09T13:46:02.793972Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28260991}
2024-08-09T16:46:26.378758113+03:00 stderr F {"level":"info","ts":"2024-08-09T13:46:26.378595Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001ae8ab2.snap"}
2024-08-09T16:49:11.508292387+03:00 stderr F {"level":"info","ts":"2024-08-09T13:49:11.508114Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47858121}
2024-08-09T16:49:11.571569269+03:00 stderr F {"level":"info","ts":"2024-08-09T13:49:11.571228Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47858121,"took":"62.078263ms","hash":879002936}
2024-08-09T16:49:11.57160287+03:00 stderr F {"level":"info","ts":"2024-08-09T13:49:11.5713Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":879002936,"revision":47858121,"compact-revision":47855848}
2024-08-09T16:54:11.515221995+03:00 stderr F {"level":"info","ts":"2024-08-09T13:54:11.514885Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47860391}
2024-08-09T16:54:11.578597578+03:00 stderr F {"level":"info","ts":"2024-08-09T13:54:11.578451Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47860391,"took":"62.518369ms","hash":1620712318}
2024-08-09T16:54:11.578626278+03:00 stderr F {"level":"info","ts":"2024-08-09T13:54:11.578495Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1620712318,"revision":47860391,"compact-revision":47858121}
2024-08-09T16:59:11.522451407+03:00 stderr F {"level":"info","ts":"2024-08-09T13:59:11.522279Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47862667}
2024-08-09T16:59:11.585025477+03:00 stderr F {"level":"info","ts":"2024-08-09T13:59:11.584875Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47862667,"took":"61.408753ms","hash":1685600439}
2024-08-09T16:59:11.585055778+03:00 stderr F {"level":"info","ts":"2024-08-09T13:59:11.584939Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1685600439,"revision":47862667,"compact-revision":47860391}
2024-08-09T17:04:11.530861537+03:00 stderr F {"level":"info","ts":"2024-08-09T14:04:11.530469Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47864948}
2024-08-09T17:04:11.590813667+03:00 stderr F {"level":"info","ts":"2024-08-09T14:04:11.590647Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47864948,"took":"58.795612ms","hash":1201217355}
2024-08-09T17:04:11.590853568+03:00 stderr F {"level":"info","ts":"2024-08-09T14:04:11.590696Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1201217355,"revision":47864948,"compact-revision":47862667}
2024-08-09T17:06:23.283023184+03:00 stderr F {"level":"info","ts":"2024-08-09T14:06:23.282835Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28275992,"local-member-snapshot-index":28265991,"local-member-snapshot-count":10000}
2024-08-09T17:06:23.288659671+03:00 stderr F {"level":"info","ts":"2024-08-09T14:06:23.288558Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28275992}
2024-08-09T17:06:23.288743572+03:00 stderr F {"level":"info","ts":"2024-08-09T14:06:23.288663Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28270992}
2024-08-09T17:06:26.406798647+03:00 stderr F {"level":"info","ts":"2024-08-09T14:06:26.406659Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001aeb1c3.snap"}
2024-08-09T17:09:11.536908146+03:00 stderr F {"level":"info","ts":"2024-08-09T14:09:11.536721Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47867221}
2024-08-09T17:09:11.602059056+03:00 stderr F {"level":"info","ts":"2024-08-09T14:09:11.601868Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47867221,"took":"64.099994ms","hash":1227210972}
2024-08-09T17:09:11.602101057+03:00 stderr F {"level":"info","ts":"2024-08-09T14:09:11.60192Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1227210972,"revision":47867221,"compact-revision":47864948}
2024-08-09T17:14:11.543750962+03:00 stderr F {"level":"info","ts":"2024-08-09T14:14:11.543573Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47869497}
2024-08-09T17:14:11.604641107+03:00 stderr F {"level":"info","ts":"2024-08-09T14:14:11.604274Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47869497,"took":"59.747827ms","hash":1948568407}
2024-08-09T17:14:11.604676107+03:00 stderr F {"level":"info","ts":"2024-08-09T14:14:11.604324Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1948568407,"revision":47869497,"compact-revision":47867221}
2024-08-09T17:19:11.551409091+03:00 stderr F {"level":"info","ts":"2024-08-09T14:19:11.551199Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47871769}
2024-08-09T17:19:11.615065979+03:00 stderr F {"level":"info","ts":"2024-08-09T14:19:11.614913Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47871769,"took":"62.801374ms","hash":2616564623}
2024-08-09T17:19:11.615097979+03:00 stderr F {"level":"info","ts":"2024-08-09T14:19:11.614962Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2616564623,"revision":47871769,"compact-revision":47869497}
2024-08-09T17:24:11.55813742+03:00 stderr F {"level":"info","ts":"2024-08-09T14:24:11.557969Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47874048}
2024-08-09T17:24:11.620082681+03:00 stderr F {"level":"info","ts":"2024-08-09T14:24:11.619743Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47874048,"took":"60.824344ms","hash":2299169128}
2024-08-09T17:24:11.620135682+03:00 stderr F {"level":"info","ts":"2024-08-09T14:24:11.61979Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2299169128,"revision":47874048,"compact-revision":47871769}
2024-08-09T17:26:43.467169168+03:00 stderr F {"level":"info","ts":"2024-08-09T14:26:43.467031Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28285993,"local-member-snapshot-index":28275992,"local-member-snapshot-count":10000}
2024-08-09T17:26:43.47113363+03:00 stderr F {"level":"info","ts":"2024-08-09T14:26:43.471034Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28285993}
2024-08-09T17:26:43.47116673+03:00 stderr F {"level":"info","ts":"2024-08-09T14:26:43.471131Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28280993}
2024-08-09T17:26:56.438103411+03:00 stderr F {"level":"info","ts":"2024-08-09T14:26:56.437949Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001aed8d4.snap"}
2024-08-09T17:29:11.565098488+03:00 stderr F {"level":"info","ts":"2024-08-09T14:29:11.564917Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47876324}
2024-08-09T17:29:11.627801661+03:00 stderr F {"level":"info","ts":"2024-08-09T14:29:11.627638Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47876324,"took":"61.839059ms","hash":290003497}
2024-08-09T17:29:11.627890262+03:00 stderr F {"level":"info","ts":"2024-08-09T14:29:11.627691Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":290003497,"revision":47876324,"compact-revision":47874048}
2024-08-09T17:34:11.571154042+03:00 stderr F {"level":"info","ts":"2024-08-09T14:34:11.570975Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47878595}
2024-08-09T17:34:11.631754282+03:00 stderr F {"level":"info","ts":"2024-08-09T14:34:11.631581Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47878595,"took":"59.728527ms","hash":1636624515}
2024-08-09T17:34:11.631801682+03:00 stderr F {"level":"info","ts":"2024-08-09T14:34:11.631631Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1636624515,"revision":47878595,"compact-revision":47876324}
2024-08-09T17:39:11.576788589+03:00 stderr F {"level":"info","ts":"2024-08-09T14:39:11.576609Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47880870}
2024-08-09T17:39:11.644298036+03:00 stderr F {"level":"info","ts":"2024-08-09T14:39:11.644075Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47880870,"took":"66.459631ms","hash":2177786861}
2024-08-09T17:39:11.644333337+03:00 stderr F {"level":"info","ts":"2024-08-09T14:39:11.644134Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2177786861,"revision":47880870,"compact-revision":47878595}
2024-08-09T17:44:11.583371244+03:00 stderr F {"level":"info","ts":"2024-08-09T14:44:11.58319Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47883149}
2024-08-09T17:44:11.645742211+03:00 stderr F {"level":"info","ts":"2024-08-09T14:44:11.645584Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47883149,"took":"61.454354ms","hash":4258833988}
2024-08-09T17:44:11.645776912+03:00 stderr F {"level":"info","ts":"2024-08-09T14:44:11.64563Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":4258833988,"revision":47883149,"compact-revision":47880870}
2024-08-09T17:47:04.352600635+03:00 stderr F {"level":"info","ts":"2024-08-09T14:47:04.352425Z","caller":"etcdserver/server.go:1395","msg":"triggering snapshot","local-member-id":"86e02918fafb02ec","local-member-applied-index":28295994,"local-member-snapshot-index":28285993,"local-member-snapshot-count":10000}
2024-08-09T17:47:04.358457226+03:00 stderr F {"level":"info","ts":"2024-08-09T14:47:04.357931Z","caller":"etcdserver/server.go:2415","msg":"saved snapshot","snapshot-index":28295994}
2024-08-09T17:47:04.358551528+03:00 stderr F {"level":"info","ts":"2024-08-09T14:47:04.35805Z","caller":"etcdserver/server.go:2445","msg":"compacted Raft logs","compact-index":28290994}
2024-08-09T17:47:26.464752854+03:00 stderr F {"level":"info","ts":"2024-08-09T14:47:26.464619Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/000000000000000f-0000000001aeffe5.snap"}
2024-08-09T17:49:11.591945032+03:00 stderr F {"level":"info","ts":"2024-08-09T14:49:11.591765Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47885422}
2024-08-09T17:49:11.65497781+03:00 stderr F {"level":"info","ts":"2024-08-09T14:49:11.654836Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47885422,"took":"62.043763ms","hash":1250837197}
2024-08-09T17:49:11.65501111+03:00 stderr F {"level":"info","ts":"2024-08-09T14:49:11.65488Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1250837197,"revision":47885422,"compact-revision":47883149}
2024-08-09T17:54:11.599137499+03:00 stderr F {"level":"info","ts":"2024-08-09T14:54:11.598883Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47887696}
2024-08-09T17:54:11.660300148+03:00 stderr F {"level":"info","ts":"2024-08-09T14:54:11.660151Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47887696,"took":"60.214834ms","hash":2419204901}
2024-08-09T17:54:11.660333848+03:00 stderr F {"level":"info","ts":"2024-08-09T14:54:11.660203Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":2419204901,"revision":47887696,"compact-revision":47885422}
2024-08-09T17:55:19.391700048+03:00 stderr F {"level":"warn","ts":"2024-08-09T14:55:19.391482Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"132.405755ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"range_response_count:0 size:7"}
2024-08-09T17:55:19.391750149+03:00 stderr F {"level":"warn","ts":"2024-08-09T14:55:19.391541Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"125.088541ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/cattle-controllers\" ","response":"range_response_count:1 size:467"}
2024-08-09T17:55:19.391758449+03:00 stderr F {"level":"info","ts":"2024-08-09T14:55:19.391574Z","caller":"traceutil/trace.go:171","msg":"trace[1477837598] range","detail":"{range_begin:/registry/health; range_end:; response_count:0; response_revision:47890487; }","duration":"132.502256ms","start":"2024-08-09T14:55:19.259052Z","end":"2024-08-09T14:55:19.391555Z","steps":["trace[1477837598] 'range keys from in-memory index tree'  (duration: 130.770529ms)"],"step_count":1}
2024-08-09T17:55:19.39180805+03:00 stderr F {"level":"info","ts":"2024-08-09T14:55:19.391705Z","caller":"traceutil/trace.go:171","msg":"trace[729564950] range","detail":"{range_begin:/registry/leases/kube-system/cattle-controllers; range_end:; response_count:1; response_revision:47890487; }","duration":"125.148441ms","start":"2024-08-09T14:55:19.266435Z","end":"2024-08-09T14:55:19.391584Z","steps":["trace[729564950] 'range keys from in-memory index tree'  (duration: 123.318213ms)"],"step_count":1}
2024-08-09T17:59:11.606169348+03:00 stderr F {"level":"info","ts":"2024-08-09T14:59:11.605941Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47889968}
2024-08-09T17:59:11.669050624+03:00 stderr F {"level":"info","ts":"2024-08-09T14:59:11.66891Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":47889968,"took":"61.928161ms","hash":3254829352}
2024-08-09T17:59:11.669092825+03:00 stderr F {"level":"info","ts":"2024-08-09T14:59:11.668959Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":3254829352,"revision":47889968,"compact-revision":47887696}
2024-08-09T18:04:11.613525372+03:00 stderr F {"level":"info","ts":"2024-08-09T15:04:11.613326Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":47892246}
2024-08-09T18:04:11.673891508+03:00 stderr F {"level":"info","ts":"2024-08-09T15:04:11.673731Z","calle

brandond Aug 14, 2024
Maintainer

Aug 09 14:52:52 rke21 systemd[1]: Stopping Rancher Kubernetes Engine v2 (server)...
Aug 09 14:52:52 rke21 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Aug 09 14:52:52 rke21 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Aug 09 14:52:52 rke21 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Aug 09 14:52:52 rke21 systemd[1]: rke2-server.service: Consumed 2h 14min 13.756s CPU time.
Aug 09 15:00:02 rke21 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...

You stopped it, and left it down for almost 8 minutes, before starting it again. The fact that it exited with a non-zero code when stopped does not indicate a problem, that is normal.

This doesn't look like a crash to me?

cengizhanR Aug 19, 2024
Author

Okey, Thank you very much for your interest. It may be our mistake. @brandond

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yesterday, one of my master nodes crashed. I am sharing the Kuebelet logs. Can you help? RKE2-server status failed. fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" #6529

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Yesterday, one of my master nodes crashed. I am sharing the Kuebelet logs. Can you help? RKE2-server status failed. fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" #6529

cengizhanR Aug 10, 2024

Replies: 2 comments · 6 replies

cengizhanR Aug 10, 2024 Author

brandond Aug 14, 2024 Maintainer

brandond Aug 14, 2024 Maintainer

cengizhanR Aug 14, 2024 Author

cengizhanR Aug 14, 2024 Author

brandond Aug 14, 2024 Maintainer

cengizhanR Aug 19, 2024 Author

cengizhanR
Aug 10, 2024

Replies: 2 comments 6 replies

cengizhanR
Aug 10, 2024
Author

brandond
Aug 14, 2024
Maintainer

brandond Aug 14, 2024
Maintainer

cengizhanR Aug 14, 2024
Author

cengizhanR Aug 14, 2024
Author

brandond Aug 14, 2024
Maintainer

cengizhanR Aug 19, 2024
Author