Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Bucket Lock immutable objects support for AWS S3 #818

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ishan16696
Copy link
Member

@ishan16696 ishan16696 commented Dec 24, 2024

What this PR does / why we need it:
This PR enables backup-restore to handle AWS S3's object lock mechanism which will make snapshots taken by backup-restore immutable. It also adjusted the Restoration and garbage collector functionality to handle immutable snapshots for S3 object store.

Which issue(s) this PR fixes:
Fixes #777

Special notes for your reviewer:
These case I have tested:

For Restoration functionality

  • Restoration functionality of backup-restore should work with buckets which don't have bucket/object lock enabled (Backward compatibility).
  • Restoration functionality of backup-restore should work with buckets which have both versioned immutable as well as non-immutable, non-versioned snapshots present.
  • Restoration functionality of backup-restore should work with buckets which have only immutable versioned snapshots(full as well as delta) present.

For Garbage collection of snapshots:

  • Backward compatibility with buckets which don't have immutability enabled.
  • GC should work with buckets which have both immutable as well as non immutable objects.
  • GC should work with buckets which have only immutable objects.

Release note:

etcd-backup-restore now supports immutable objects for storage provider: AWS S3, provided by the [Bucket Lock](https://aws.amazon.com/s3/features/object-lock/) feature.
Snapshots garbage collection performed by etcd-backup-restore (if enabled) for S3 is performed only when the snapshots's retention period get expires.
Support for Bucket lock (S3) in etcd-backup-restore is backward compatible. For more info please refer to this doc: https://github.com/gardener/etcd-backup-restore/blob/master/docs/usage/enabling_immutable_snapshots.md

@ishan16696 ishan16696 requested a review from a team as a code owner December 24, 2024 05:45
@gardener-robot gardener-robot added needs/review Needs review needs/rebase Needs git rebase labels Dec 24, 2024
@gardener-robot
Copy link

@ishan16696 You need rebase this pull request with latest master branch. Please check.

@gardener-robot gardener-robot added the size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) label Dec 24, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 24, 2024
@shreyas-s-rao shreyas-s-rao added this to the v0.33.0 milestone Dec 24, 2024
@gardener-robot gardener-robot added size/l Size of pull request is large (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else and removed size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) labels Dec 25, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 25, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 25, 2024
@gardener-robot gardener-robot added size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) and removed size/l Size of pull request is large (see gardener-robot robot/bots/size.py) labels Dec 25, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 25, 2024
@gardener-robot gardener-robot added size/l Size of pull request is large (see gardener-robot robot/bots/size.py) and removed size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) labels Dec 27, 2024
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 27, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 27, 2024
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 27, 2024
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Dec 27, 2024
@renormalize renormalize self-assigned this Dec 30, 2024
Copy link
Collaborator

@shreyas-s-rao shreyas-s-rao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ishan16696 thanks for the well-written PR and the docs. I have a few nits. PTAL, thanks.

docs/usage/enabling_immutable_snapshots.md Outdated Show resolved Hide resolved
docs/usage/enabling_immutable_snapshots.md Outdated Show resolved Hide resolved
pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved
pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved

if *objectConfig.ObjectLockConfiguration.ObjectLockEnabled == "Enabled" {
// assumption: retention period of bucket will always be in days, not years.
return true, objectConfig.ObjectLockConfiguration.Rule.DefaultRetention.Days, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each of the fields in objectConfig.ObjectLockConfiguration.Rule.DefaultRetention.Days is of pointer type, so again possibility of nil pointer dereference.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here, it won't be an issue as caller for this func is checking the nil pointer reference.

if bucketImmutableExpiryTimeInDays != nil {
....
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bucketImmutableExpiryTimeInDays is deduced from GetBucketImmutabilityTime(), which in turn returns objectConfig.ObjectLockConfiguration.Rule.DefaultRetention.Days. Simply checking bucketImmutableExpiryTimeInDays will not help, since you could have Rule an non-nil, but DefaultRetention could be nil, and throw an exception. That's the reason the entire path should be checked here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, got your point. I will make the change.

pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved
pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved
snap.ImmutabilityExpiryTime = snap.CreatedOn.Add(time.Duration(*bucketImmutableExpiryTimeInDays) * 24 * time.Hour)
} else {
// retry to get bucketImmutableExpiryTimeInDays
_, bucketImmutableExpiryTimeInDays, err = GetBucketImmutabilityTime(s)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is bucketImmutableExpiryTimeInDays being used here? The snapshot gets added to the snapList without setting snap.ImmutabilityExpiryTime, which doesn't seem right. Can you please confirm?

Copy link
Member Author

@ishan16696 ishan16696 Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, you are correct. Actually I added this to handle the case when previous api call failed to get bucketImmutableExpiryTimeInDays.
Another way to handle this is to add a retry mechanism in first api call, and remove this else handling part , wdyt ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, retry mechanism seems more intuitive here.

pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add some tests specific to handling immutable backups for S3 buckets? Like garbage collection for instance.

@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jan 3, 2025
…l make snapshots immutable.

Adjusted the Restoration and GC functionality to handle immutable snapshots for S3 object store.
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 3, 2025
@ishan16696
Copy link
Member Author

I have rebased the PR on master.

@gardener-robot-ci-1 gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/rebase Needs git rebase needs/review Needs review needs/second-opinion Needs second review by someone else size/l Size of pull request is large (see gardener-robot robot/bots/size.py)
Projects
None yet
9 participants