Add `Container level WORM` immutable object support for Azure Blob Storage. #781

renormalize · 2024-09-30T05:33:54Z

What this PR does / why we need it:

Enables etcd-backup-restore to handle immutable objects as provided by Container Level WORM in Azure Blob Storage.

This PR mirrors #776 for Azure Blob Storage.

Which issue(s) this PR fixes:
Fixes # #777 only for ABS.

Special notes for your reviewer:

The same conventions are followed as in #776. Please let me know if you find anything that doesn't stick to those conventions.

Release note:

etcd-backup-restore now supports immutable objects for storage provider: Azure Blob Storage, provided by the [Container Level WORM](hhttps://learn.microsoft.com/en-us/azure/storage/blobs/immutable-container-level-worm-policies) feature.

Snapshots garbage collection performed by etcd-backup-restore (if enabled) for ABS is performed only when the objects' immutability period expires.

Support for Container level WORM (ABS) in etcd-backup-restore is backward compatible. For more info please refer to this doc: https://github.com/gardener/etcd-backup-restore/blob/master/docs/usage/immutable_snapshots.md

anveshreddy18 · 2024-10-03T12:09:03Z

/assign

anveshreddy18

Thanks for the PR @renormalize. I have few comments, PTAL

pkg/snapstore/abs_snapstore.go

pkg/snapstore/abs_snapstore_test.go

anveshreddy18

Thanks for the PR, LGTM!

just one small mention: As discussed, it would be better if we mention somewhere that adding tags is only possible through CLI at the moment.

ishan16696

overall looks good, few nits

pkg/snapstore/abs_snapstore.go

renormalize · 2024-11-19T18:11:44Z

@ishan16696 yes I'll get to it ASAP.

…ABS.

* `List` is enhanced to optionally return tagged snapshots. * Unit tests for tagged snapshots added.

…he `"x-etcd-snapshot-exclude"="true"` tag.

renormalize · 2024-11-22T10:48:27Z

Needs changes. Please do not merge.

renormalize · 2024-11-25T10:14:11Z

pkg/types/snapstore.go

+var (
+	// ErrSnapshotDeleteFailDueToImmutability is the error returned when the Delete call fails due to immutability
+	ErrSnapshotDeleteFailDueToImmutability = fmt.Errorf("ErrSnapshotDeleteFailDueToImmutability")
+)


This is the easiest way to establish a custom error for garbage collection, for dealing with errors faced during deletion of immutable objects.
In the future this should be removed, and a more comprehensive custom error should be created.

pkg/snapshot/snapshotter/garbagecollector.go

ishan16696

LGTM!!

renormalize added area/robustness Robustness, reliability, resilience related area/backup Backup related area/storage Storage related platform/azure Microsoft Azure platform/infrastructure labels Sep 30, 2024

renormalize added this to the 2024-Q4 milestone Sep 30, 2024

renormalize self-assigned this Sep 30, 2024

renormalize requested a review from a team as a code owner September 30, 2024 05:33

gardener-robot added the needs/review Needs review label Sep 30, 2024

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 30, 2024

gardener-robot added the size/s Size of pull request is small (see gardener-robot robot/bots/size.py) label Sep 30, 2024

gardener-robot-ci-2 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 30, 2024

ishan16696 self-assigned this Oct 3, 2024

gardener-robot assigned anveshreddy18 Oct 3, 2024

anveshreddy18 requested changes Oct 7, 2024

View reviewed changes

gardener-robot added the needs/changes Needs (more) changes label Oct 7, 2024

shreyas-s-rao modified the milestones: 2024-Q4, v0.31.0 Oct 8, 2024

gardener-robot added size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) and removed size/s Size of pull request is small (see gardener-robot robot/bots/size.py) labels Oct 8, 2024

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 8, 2024

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 8, 2024

renormalize force-pushed the immutable-abs branch from ba302f7 to ff22e90 Compare October 8, 2024 14:44

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Oct 8, 2024

ishan16696 mentioned this pull request Oct 10, 2024

[Feature] ☂️ Support Bucket Lock mechanism for backups taken by backup-restore for various storage provider #777

Open

3 tasks

anveshreddy18 approved these changes Oct 16, 2024

View reviewed changes

ishan16696 requested changes Oct 16, 2024

View reviewed changes

pkg/snapstore/abs_snapstore.go Outdated Show resolved Hide resolved

pkg/snapstore/abs_snapstore.go Outdated Show resolved Hide resolved

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Oct 18, 2024

renormalize added 7 commits November 22, 2024 10:42

Skip tagged blobs, and add retention expiry data in the snapshot for …

d9cde02

…ABS.

Fix existing unit tests for ABS.

276572a

Enhance List to be able to optionally include tagged snapshots.

eebe680

* `List` is enhanced to optionally return tagged snapshots. * Unit tests for tagged snapshots added.

Update docs to convey immutable snapshot support in Azure Blob Storage.

f7ffa83

Address review comments from @ishan16696 1.

0770403

Add documentation about ignoring snapshots from restoration through t…

6399191

…he `"x-etcd-snapshot-exclude"="true"` tag.

Expand on documentation for ignoring snapshots.

ec870b0

renormalize force-pushed the immutable-abs branch from c3c906a to ec870b0 Compare November 22, 2024 05:48

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 22, 2024

renormalize added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 22, 2024

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 22, 2024

renormalize added the reviewed/do-not-merge Has no approval for merging as it may break things, be of poor quality or have (ext.) dependencies label Nov 22, 2024

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 25, 2024

renormalize force-pushed the immutable-abs branch from 3b14cc3 to f6a9238 Compare November 25, 2024 09:26

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 25, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 25, 2024

ishan16696 removed the reviewed/do-not-merge Has no approval for merging as it may break things, be of poor quality or have (ext.) dependencies label Nov 25, 2024

renormalize commented Nov 25, 2024

View reviewed changes

ishan16696 reviewed Nov 25, 2024

View reviewed changes

pkg/snapshot/snapshotter/garbagecollector.go Show resolved Hide resolved

Handle Immutability error responses from the Azure Blob Storage API.

5f6832c

renormalize force-pushed the immutable-abs branch from f6a9238 to 5f6832c Compare November 25, 2024 10:59

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 25, 2024

ishan16696 approved these changes Nov 25, 2024

View reviewed changes

renormalize merged commit 1936760 into gardener:master Nov 25, 2024
9 checks passed

gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Nov 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Container level WORM` immutable object support for Azure Blob Storage. #781

Add `Container level WORM` immutable object support for Azure Blob Storage. #781

renormalize commented Sep 30, 2024

anveshreddy18 commented Oct 3, 2024

anveshreddy18 left a comment

anveshreddy18 left a comment

ishan16696 left a comment

renormalize commented Nov 19, 2024

renormalize commented Nov 22, 2024

renormalize Nov 25, 2024

ishan16696 left a comment

Add Container level WORM immutable object support for Azure Blob Storage. #781

Add Container level WORM immutable object support for Azure Blob Storage. #781

Conversation

renormalize commented Sep 30, 2024

anveshreddy18 commented Oct 3, 2024

anveshreddy18 left a comment

Choose a reason for hiding this comment

anveshreddy18 left a comment

Choose a reason for hiding this comment

ishan16696 left a comment

Choose a reason for hiding this comment

renormalize commented Nov 19, 2024

renormalize commented Nov 22, 2024

renormalize Nov 25, 2024

Choose a reason for hiding this comment

ishan16696 left a comment

Choose a reason for hiding this comment

Add `Container level WORM` immutable object support for Azure Blob Storage. #781

Add `Container level WORM` immutable object support for Azure Blob Storage. #781