Enables backup-restore to handle S3's object lock mechanism which wil…

…l make snapshots immutable.

Adjusted the Restoration and GC functionality to handle immutable snapshots for S3 object store.

docs/usage/immutable_snapshots.md

@@ -8,29 +8,57 @@ Currently, etcd-backup-restore supports the use of immutable objects on the foll
 
 - Google Cloud Storage
 - Azure Blob Storage
+- AWS S3
 
 ## Enabling and using Immutable Snapshots with etcd-backup-restore
 
 Etcd-backup-restore supports immutable objects, typically at what cloud providers call the "bucket level." During the creation of a bucket, it is configured to render objects immutable for a specific duration from the moment of their upload. This feature can be enabled through:
 
 - **Google Cloud Storage**: [Bucket Lock](https://cloud.google.com/storage/docs/bucket-lock)
 - **Azure Blob Storage**: [Container-level WORM Policies](https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-container-level-worm-policies)
+- **AWS S3**: [S3 object Lock](https://aws.amazon.com/s3/features/object-lock/)
 
 It is also possible to enable immutability retroactively by making appropriate API calls to your cloud provider, allowing the immutable snapshots feature to be used with existing buckets. For information on such configurations, please refer to your cloud provider's documentation.
 
 Setting the immutability period at the bucket level through the CLI can be done through the following commands:
 
 - Google Cloud Storage would be done like so:
 
-    ```sh
-    gcloud storage buckets update <your-bucket> --retention-period=<desired-immutability-period>
-    ```
+```sh
+gcloud storage buckets update <your-bucket> --retention-period=<desired-immutability-period>
+```
 
 - Azure Blob Storage Container would be done like so:
 
-    ```sh
-    az storage container immutability-policy create --resource-group <your-resource-group> --account-name <your-account-name> --container-name <your-container-name> --period <desired-immutability-period>
-    ```
+```sh
+az storage container immutability-policy create --resource-group <your-resource-group> --account-name <your-account-name> --container-name <your-container-name> --period <desired-immutability-period>
+```
+
+- AWS S3:
+
+  1. To enable the object lock on new buckets
+
+   * Create a new bucket with object lock then update the bucket with object lock configuration.
+
+```bash
+# create new bucket with object lock enabled
+aws s3api create-bucket --bucket <your-bucket-name> --region <region> --create-bucket-configuration LocationConstraint=<region> --object-lock-enabled-for-bucket
+
+# update the bucket with object lock configuration
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
+
+  2. To enable the object lock on existing buckets
+
+  * First enable the object versioning on existing bucket then enable the object lock on bucket with its configurations (say `X` retention period).
+
+```bash
+# enable the object versioning on existing bucket
+aws s3api put-bucket-versioning --bucket <your-bucket-name> --versioning-configuration Status=Enabled
+
+# now, enable the object lock on bucket with its configurations
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
 
 The behaviour of bucket's objects uploaded before a bucket is set to immutable varies among storage providers. etcd-backup-restore manages these objects and will perform garbage collection according to the configured garbage collection policy and the object's immutability expiry.
 
@@ -80,3 +108,29 @@ You can add these tags through for the following providers like so:
     Use the Azure Portal to add the tag in the `Blob index tags` section of the blob.
 
 Once these annotations/tags are added, etcd-backup-restore will ignore those snapshots during restoration.
+
+- **AWS S3**:
+
+  - This method of tagging the snapshots to skip any snapshots during restoration is not supported for `AWS S3` buckets.
+  - For object lock, S3 object versioning will automatically get enabled. So this extra handling is not required as user can simply soft delete those snapshots.
+  - With object versioning inplace, a deletion marker will get added on top of those snapshots, and during the restoration of backup-restore, it will only considers the latest snapshots.
+  - For more info: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html
+
+### S3 Object Lock and working with snapshots
+
+#### Object Lock
+
+- S3 Object Lock blocks permanent object deletion for a user defined retention period.
+- It works on WORM(write once read many) model.
+- With S3 object lock, S3 versioning will automatically get enabled, it only prevent locked object versions from being permanently deleted.
+
+   > Note: The consumer of etcd-backup-restore must enable the object lock with the appropriate settings on their buckets to consume this feature. This is because backup-restore doesn't manage or interfere with the bucket (object store) creation process.
+
+#### Working with snapshots
+
+- S3 Object Lock can be activated at either on the bucket or object level. Moreover, it can be enabled when creating a new buckets or on a existing buckets.
+- For new buckets: These buckets will only contains the new snapshots, hence all the snapshots inside this bucket will be versioned locked snapshots.
+- For existing/old buckets: These buckets can contain a mix of pre-existing non-versioned, non-locked snapshots and new snapshots which are versioned and locked with retention period.
+The following diagram illustrates the working of snapshots with S3 for existing/old buckets as well as for new buckets.
+
+  ![Working with S3](../images/S3_immutability_working.png)

pkg/snapshot/snapshotter/garbagecollector.go

@@ -181,6 +181,10 @@ func (ssr *Snapshotter) RunGarbageCollector(stopCh <-chan struct{}) {
 					if fullSnapshotIndex < len(fullSnapshotIndexList)-int(ssr.config.MaxBackups) {
 						snap := snapList[fullSnapshotIndexList[fullSnapshotIndex]]
 						snapPath := path.Join(snap.SnapDir, snap.SnapName)
+						if !snap.IsDeletable() {
+							ssr.logger.Infof("GC: Skipping the snapshot: %s, since its immutability period hasn't expired yet", snap.SnapName)
+							continue
+						}
 						ssr.logger.Infof("GC: Deleting old full snapshot: %s", snapPath)
 						if err := ssr.store.Delete(*snap); errors.Is(err, brtypes.ErrSnapshotDeleteFailDueToImmutability) {
 							// The snapshot is still immutable, attempt to gargbage collect it in the next run

pkg/snapstore/oss_snapstore.go

@@ -52,7 +52,6 @@ type authOptions struct {
 type OSSSnapStore struct {
 	prefix                  string
 	bucket                  OSSBucket
-	multiPart               sync.Mutex
 	maxParallelChunkUploads uint
 	minChunkSize            int64
 	tempDir                 string

pkg/snapstore/s3_snapstore.go

@@ -64,10 +64,9 @@ type SSECredentials struct {
 
 // S3SnapStore is snapstore with AWS S3 object store as backend
 type S3SnapStore struct {
-	prefix    string
-	client    s3iface.S3API
-	bucket    string
-	multiPart sync.Mutex
+	prefix string
+	client s3iface.S3API
+	bucket string
 	// maxParallelChunkUploads hold the maximum number of parallel chunk uploads allowed.
 	maxParallelChunkUploads uint
 	minChunkSize            int64
@@ -137,7 +136,7 @@ func readAWSCredentialsJSONFile(filename string) (session.Options, SSECredential
 	}
 
 	httpClient := http.DefaultClient
-	if awsConfig.InsecureSkipVerify != nil && *awsConfig.InsecureSkipVerify == true {
+	if awsConfig.InsecureSkipVerify != nil && *awsConfig.InsecureSkipVerify {
 		httpClient.Transport = &http.Transport{
 			TLSClientConfig: &tls.Config{InsecureSkipVerify: *awsConfig.InsecureSkipVerify},
 		}
@@ -191,7 +190,7 @@ func readAWSCredentialFiles(dirname string) (session.Options, SSECredentials, er
 	}
 
 	httpClient := http.DefaultClient
-	if awsConfig.InsecureSkipVerify != nil && *awsConfig.InsecureSkipVerify == true {
+	if awsConfig.InsecureSkipVerify != nil && *awsConfig.InsecureSkipVerify {
 		httpClient.Transport = &http.Transport{
 			TLSClientConfig: &tls.Config{InsecureSkipVerify: *awsConfig.InsecureSkipVerify},
 		}
@@ -320,9 +319,18 @@ func NewS3FromClient(bucket, prefix, tempDir string, maxParallelChunkUploads uin
 
 // Fetch should open reader for the snapshot file from store
 func (s *S3SnapStore) Fetch(snap brtypes.Snapshot) (io.ReadCloser, error) {
-	getObjectInput := &s3.GetObjectInput{
-		Bucket: aws.String(s.bucket),
-		Key:    aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
+	getObjectInput := &s3.GetObjectInput{}
+	if len(snap.VersionID) > 0 {
+		getObjectInput = &s3.GetObjectInput{
+			Bucket:    aws.String(s.bucket),
+			Key:       aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
+			VersionId: &snap.VersionID,
+		}
+	} else {
+		getObjectInput = &s3.GetObjectInput{
+			Bucket: aws.String(s.bucket),
+			Key:    aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
+		}
 	}
 	if s.sseCustomerKey != "" {
 		// Customer managed Server Side Encryption
@@ -511,34 +519,98 @@ func (s *S3SnapStore) partUploader(wg *sync.WaitGroup, stopCh <-chan struct{}, s
 }
 
 // List will return sorted list with all snapshot files on store.
-func (s *S3SnapStore) List(_ bool) (brtypes.SnapList, error) {
+// For S3 object List will return the list of all
+func (s *S3SnapStore) List(includeAll bool) (brtypes.SnapList, error) {
+	var snapList brtypes.SnapList
 	prefixTokens := strings.Split(s.prefix, "/")
 	// Last element of the tokens is backup version
 	// Consider the parent of the backup version level (Required for Backward Compatibility)
 	prefix := path.Join(strings.Join(prefixTokens[:len(prefixTokens)-1], "/"))
 
-	var snapList brtypes.SnapList
-	in := &s3.ListObjectsInput{
-		Bucket: aws.String(s.bucket),
-		Prefix: aws.String(prefix),
-	}
-	err := s.client.ListObjectsPages(in, func(page *s3.ListObjectsOutput, lastPage bool) bool {
-		for _, key := range page.Contents {
-			k := (*key.Key)[len(*page.Prefix):]
-			if strings.Contains(k, backupVersionV1) || strings.Contains(k, backupVersionV2) {
-				snap, err := ParseSnapshot(path.Join(prefix, k))
-				if err != nil {
-					// Warning
-					logrus.Warnf("Invalid snapshot found. Ignoring it: %s", k)
+	// Get the status of bucket versioning.
+	// Note: Bucket versioning will always be enabled for object lock.
+	versioningStatus, err := s.client.GetBucketVersioning(&s3.GetBucketVersioningInput{Bucket: &s.bucket})
+	if err != nil {
+		return nil, err
+	}
+
+	if versioningStatus.Status != nil && *versioningStatus.Status == "Enabled" {
+		// object/bucket versioning is found to be enabled on given bucket.
+		logrus.Info("Object versioning is found to be enabled.")
+
+		isObjectLockEnabled, bucketImmutableExpiryTimeInDays, err := getBucketImmutabilityTime(s)
+		if err != nil {
+			logrus.Warnf("unable to check object lock configuration for the bucket: %v", err)
+		} else if !isObjectLockEnabled {
+			logrus.Warnf("Object versioning is found to be enabled but object lock is not found to be enabled.")
+			logrus.Warnf("Please enable the object lock as well for the given bucket for immutability of snapshots.")
+		}
+
+		in := &s3.ListObjectVersionsInput{
+			Bucket: aws.String(s.bucket),
+			Prefix: aws.String(prefix),
+		}
+
+		if err := s.client.ListObjectVersionsPages(in, func(page *s3.ListObjectVersionsOutput, lastPage bool) bool {
+			for _, version := range page.Versions {
+				if *version.IsLatest {
+					k := (*version.Key)[len(*page.Prefix):]
+					if strings.Contains(k, backupVersionV1) || strings.Contains(k, backupVersionV2) {
+						snap, err := ParseSnapshot(path.Join(prefix, k))
+						if err != nil {
+							// Warning
+							logrus.Warnf("Invalid snapshot found. Ignoring it: %s", k)
+						} else {
+							// capture the versionID of snapshot and expiry time of snapshot
+							snap.VersionID = *version.VersionId
+							if bucketImmutableExpiryTimeInDays != nil {
+								// To get S3's object "RetainUntilDate" or "ImmutabilityExpiryTime", backup-restore need to make an API call for each snapshots.
+								// To avoid API calls for each snapshots, backup-restore is calculating the "ImmutabilityExpiryTime" using bucket retention period.
+								// ImmutabilityExpiryTime = SnapshotCreationTime + ObjectRetentionTimeInDays
+								snap.ImmutabilityExpiryTime = snap.CreatedOn.Add(time.Duration(*bucketImmutableExpiryTimeInDays) * 24 * time.Hour)
+							} else {
+								_, bucketImmutableExpiryTimeInDays, err = getBucketImmutabilityTime(s)
+								if err != nil {
+									logrus.Warnf("unable to get bucket immutability expiry time: %v", err)
+								}
+							}
+							snapList = append(snapList, snap)
+						}
+					}
 				} else {
-					snapList = append(snapList, snap)
+					// Warning
+					logrus.Warnf("Snapshot: %s with versionID: %s found to be not latest, it was last modified: %s. Ignoring it.", *version.Key, *version.VersionId, version.LastModified)
 				}
 			}
+			return !lastPage
+		}); err != nil {
+			return nil, err
+		}
+	} else {
+		// object/bucket versioning is not found to be enabled on given bucket.
+		logrus.Info("Object versioning is not found to be enabled.")
+		in := &s3.ListObjectsInput{
+			Bucket: aws.String(s.bucket),
+			Prefix: aws.String(prefix),
+		}
+
+		if err := s.client.ListObjectsPages(in, func(page *s3.ListObjectsOutput, lastPage bool) bool {
+			for _, key := range page.Contents {
+				k := (*key.Key)[len(*page.Prefix):]
+				if strings.Contains(k, backupVersionV1) || strings.Contains(k, backupVersionV2) {
+					snap, err := ParseSnapshot(path.Join(prefix, k))
+					if err != nil {
+						// Warning
+						logrus.Warnf("Invalid snapshot found. Ignoring it: %s", k)
+					} else {
+						snapList = append(snapList, snap)
+					}
+				}
+			}
+			return !lastPage
+		}); err != nil {
+			return nil, err
 		}
-		return !lastPage
-	})
-	if err != nil {
-		return nil, err
 	}
 
 	sort.Sort(snapList)
@@ -547,11 +619,25 @@ func (s *S3SnapStore) List(_ bool) (brtypes.SnapList, error) {
 
 // Delete should delete the snapshot file from store
 func (s *S3SnapStore) Delete(snap brtypes.Snapshot) error {
-	_, err := s.client.DeleteObject(&s3.DeleteObjectInput{
-		Bucket: aws.String(s.bucket),
-		Key:    aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
-	})
-	return err
+	if len(snap.VersionID) > 0 {
+		// to delete versioned snapshots present in bucket.
+		if _, err := s.client.DeleteObject(&s3.DeleteObjectInput{
+			Bucket:    aws.String(s.bucket),
+			Key:       aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
+			VersionId: &snap.VersionID,
+		}); err != nil {
+			return err
+		}
+	} else {
+		// to delete non-versioned snapshots present in bucket.
+		if _, err := s.client.DeleteObject(&s3.DeleteObjectInput{
+			Bucket: aws.String(s.bucket),
+			Key:    aws.String(path.Join(snap.Prefix, snap.SnapDir, snap.SnapName)),
+		}); err != nil {
+			return err
+		}
+	}
+	return nil
 }
 
 // GetS3CredentialsLastModifiedTime returns the latest modification timestamp of the AWS credential file(s)
@@ -622,3 +708,19 @@ func getSSECreds(sseCustomerKey, sseCustomerAlgorithm *string) (SSECredentials,
 		sseCustomerAlgorithm: *sseCustomerAlgorithm,
 	}, nil
 }
+
+func getBucketImmutabilityTime(s *S3SnapStore) (bool, *int64, error) {
+	objectConfig, err := s.client.GetObjectLockConfiguration(&s3.GetObjectLockConfigurationInput{
+		Bucket: aws.String(s.bucket),
+	})
+	if err != nil {
+		return false, nil, err
+	}
+
+	if *objectConfig.ObjectLockConfiguration.ObjectLockEnabled == "Enabled" {
+		// assumption: retention period of bucket will always be in days, not years.
+		return true, objectConfig.ObjectLockConfiguration.Rule.DefaultRetention.Days, nil
+	}
+
+	return false, nil, nil
+}

pkg/types/snapstore.go

@@ -87,7 +87,7 @@ type SnapStore interface {
 type Snapshot struct {
 	Kind                   string    `json:"kind"` // incr:incremental, full:full
 	StartRevision          int64     `json:"startRevision"`
-	LastRevision           int64     `json:"lastRevision"` // latest revision on snapshot
+	LastRevision           int64     `json:"lastRevision"` // latest revision of snapshot
 	CreatedOn              time.Time `json:"createdOn"`
 	SnapDir                string    `json:"snapDir"`
 	SnapName               string    `json:"snapName"`
@@ -96,6 +96,8 @@ type Snapshot struct {
 	CompressionSuffix      string    `json:"compressionSuffix"` // CompressionSuffix depends on compression policy
 	IsFinal                bool      `json:"isFinal"`
 	ImmutabilityExpiryTime time.Time `json:"immutabilityExpriyTime"`
+	// It is used only for AWS S3 object lock immutability.
+	VersionID string `json:"versionID"`
 }
 
 // IsDeletable determines if the snapshot can be deleted.