Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.30] - S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #11051

Closed
brandond opened this issue Oct 9, 2024 · 1 comment
Closed

[Release-1.30] - S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #11051

brandond opened this issue Oct 9, 2024 · 1 comment
Assignees
@brandond @VestigeJ
Labels
waiting-for-RC Issue is available to test only after we have an RC
Milestone
v1.30.6+k3s1

Comments

@brandond
Copy link
Member

brandond commented Oct 9, 2024

Backport fix for S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time
@brandond brandond self-assigned this Oct 9, 2024
@brandond brandond mentioned this issue Oct 10, 2024
Merged
@brandond brandond moved this from New to To Test in K3s Development Oct 10, 2024
@brandond brandond added this to the v1.30.6+k3s1 milestone Oct 10, 2024
@aganesh-suse aganesh-suse added the waiting-for-RC Issue is available to test only after we have an RC label Oct 24, 2024
@VestigeJ
Copy link

Previous behavior established here - lots of noise around etcd-snapshots taken from Rancher rancher/rke2#7027 (comment)

Screenshot 2024-10-25 at 3 22 41 PM

$ sudo kubectl get nodes

NAME                         STATUS   ROLES                              AGE     VERSION
v1306rc1-pool1-f9h45-n6x8v   Ready    control-plane,etcd,master,worker   4h28m   v1.30.6-rc1+k3s1
v1306rc1-pool1-f9h45-vzgdz   Ready    control-plane,etcd,master,worker   4h26m   v1.30.6-rc1+k3s1
v1306rc1-pool1-f9h45-zv6r8   Ready    control-plane,etcd,master,worker   4h25m   v1.30.6-rc1+k3s1
v1306rc1-pool2-5lc2q-jm9hn   Ready    worker                             4h25m   v1.30.6-rc1+k3s1

$ sudo kubectl get cm -n kube-system k3s-etcd-snapshots

NAME                 DATA   AGE
k3s-etcd-snapshots   12     141m

$ sudo k3s etcd-snapshot ls

Name                                                Location                                                                                            Size     Created
etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402 s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402           10293280 2024-10-25T20:00:02Z
etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402 10293280 2024-10-25T20:00:02Z
etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403 s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403           10166304 2024-10-25T20:00:03Z
etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403 s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403           10117152 2024-10-25T20:00:03Z
on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289     s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289               10293280 2024-10-25T22:11:29Z
on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289     10293280 2024-10-25T22:11:29Z
on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289     s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289               10166304 2024-10-25T22:11:29Z
on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289     s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289               10117152 2024-10-25T22:11:29Z

$ sudo kubectl get etcdsnapshotfile

NAME                                                               SNAPSHOTNAME                                          NODE                         LOCATION                                                                                              SIZE       CREATIONTIME
local-etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402-34087c   etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402   v1306rc1-pool1-f9h45-n6x8v   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402   10293280   2024-10-25T20:00:02Z
local-etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403-16cabc   etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403   v1306rc1-pool1-f9h45-vzgdz   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403   10166304   2024-10-25T20:00:03Z
local-etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403-2c5c4b   etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403   v1306rc1-pool1-f9h45-zv6r8   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403   10117152   2024-10-25T20:00:03Z
local-on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289-fc4ae1       on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289       v1306rc1-pool1-f9h45-n6x8v   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289       10293280   2024-10-25T22:11:29Z
local-on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289-c89d50       on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289       v1306rc1-pool1-f9h45-vzgdz   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289       10166304   2024-10-25T22:11:29Z
local-on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289-44983d       on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289       v1306rc1-pool1-f9h45-zv6r8   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289       10117152   2024-10-25T22:11:29Z
s3-etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402-e36497      etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402   s3                           s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-n6x8v-1729886402             10293280   2024-10-25T20:00:02Z
s3-etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403-ad445c      etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403   s3                           s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-vzgdz-1729886403             10166304   2024-10-25T20:00:03Z
s3-etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403-118225      etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403   s3                           s3://justin-snapshot-buckets/v1306rc1/etcd-snapshot-v1306rc1-pool1-f9h45-zv6r8-1729886403             10117152   2024-10-25T20:00:03Z
s3-on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289-046b94          on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289       s3                           s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-n6x8v-1729894289                 10293280   2024-10-25T22:11:29Z
s3-on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289-0c5450          on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289       s3                           s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-vzgdz-1729894289                 10166304   2024-10-25T22:11:29Z
s3-on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289-1f2982          on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289       s3                           s3://justin-snapshot-buckets/v1306rc1/on-demand-v1306rc1-pool1-f9h45-zv6r8-1729894289                 10117152   2024-10-25T22:11:29Z

@github-project-automation github-project-automation bot moved this from To Test to Done Issue in K3s Development Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
waiting-for-RC Issue is available to test only after we have an RC
Projects
K3s Development
Status: Done Issue
Milestone
v1.30.6+k3s1
Development

No branches or pull requests
3 participants
@brandond @VestigeJ @aganesh-suse