Enables backup-restore to handle S3's object lock mechanism which wil…

…l make snapshots immutable.

Adjusted the Restoration and GC functionality to handle immutable snapshots for S3 object store.

docs/usage/enabling_immutable_snapshots.md

@@ -1,18 +1,23 @@
 # Enabling Immutable Snapshots in `etcd-backup-restore`
 
-This guide walks you through the process of enabling immutable snapshots in `etcd-backup-restore` by leveraging bucket-level immutability features provided by cloud storage providers like Google Cloud Storage (GCS) and Azure Blob Storage (ABS). Enabling immutability ensures that your backups are tamper-proof and comply with regulatory requirements.
+This guide walks you through the process of enabling immutable snapshots in `etcd-backup-restore` by leveraging bucket-level immutability features for various object storage providers:
+
+1. Google Cloud Storage (GCS)
+2. Azure Blob Storage (ABS)
+3. Amazon Simple Storage Service (S3)
+Enabling immutability of your bucket will ensure that your backups are tamper-proof and comply with regulatory requirements.
 
 ---
 
 ## Terminology
 
-- **Bucket / Container**: A storage resource in cloud storage services where objects (such as snapshots) are stored. GCS uses the term **bucket**, while ABS uses **container**.
+- **Bucket / Container**: A storage resource in cloud storage services where objects (such as snapshots) are stored. GCS and S3 uses the term **bucket**, while ABS uses **container**.
 
-- **Immutability Policy**: A configuration that specifies a minimum period during which objects in a bucket/container are protected from deletion or modification.
+- **Immutability**: The property of an object being unmodifiable after creation, until the immutability period expires.
 
-- **Immutability Period**: The duration defined by the immutability policy during which objects remain immutable.
+- **Immutability Policy**: A configuration that specifies a minimum retention period during which objects in a bucket/container are protected from deletion or modification.
 
-- **Immutability**: The property of an object being unmodifiable after creation, until the immutability period expires.
+- **Immutability Period**: The duration defined by the immutability policy during which objects remain immutable.
 
 - **Locking**: The action of making an immutability policy permanent, preventing any reduction or removal of the immutability period.
 
@@ -22,7 +27,9 @@ This guide walks you through the process of enabling immutable snapshots in `etc
 
 ## Overview
 
-Currently, `etcd-backup-restore` supports bucket-level immutability for GCS and ABS.
+Currently, `etcd-backup-restore` supports bucket-level immutability for GCS, ABS and S3.
+
+> Note: Currently, Openstack object storage (swift) doesn't support immutability for objects: https://blueprints.launchpad.net/swift/+spec/immutability-middleware.
 
 - **Immutability Policy**: You can add an immutability policy to a bucket/container to specify an immutability period.
   - When an immutability policy is set, objects in the bucket/container can only be deleted or replaced once their age exceeds the immutability period.
@@ -37,7 +44,6 @@ Currently, `etcd-backup-restore` supports bucket-level immutability for GCS and
   - You can increase the immutability period of a locked policy if needed.
   - A locked bucket/container can only be deleted once all objects present in the bucket/container are deleted.
 
-
 ---
 
 ## Configure Bucket-Level Immutability
@@ -100,6 +106,32 @@ To configure an immutability policy on an Azure Blob Storage container:
        --period 4
    ```
 
+#### AWS S3
+
+1. To enable the object lock on new buckets
+
+   * Create a new bucket with object lock then update the bucket with object lock configuration.
+
+```bash
+# create new bucket with object lock enabled
+aws s3api create-bucket --bucket <your-bucket-name> --region <region> --create-bucket-configuration LocationConstraint=<region> --object-lock-enabled-for-bucket
+
+# update the bucket with object lock configuration
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
+
+  2. To enable the object lock on existing buckets
+
+  * First enable the object versioning on existing bucket then enable the object lock on bucket with its configurations (say `X` retention period).
+
+```bash
+# enable the object versioning on existing bucket
+aws s3api put-bucket-versioning --bucket <your-bucket-name> --versioning-configuration Status=Enabled
+
+# now, enable the object lock on bucket with its configurations
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
+
 ### Modify an Unlocked Immutability Policy
 
 You can modify an unlocked immutability policy to adjust the immutability period or to allow additional writes to the bucket/container.
@@ -199,7 +231,6 @@ To lock the immutability policy:
        --lock-retention-period
    ```
 
-
 #### Azure Blob Storage (ABS)
 
 To lock the immutability policy:
@@ -240,8 +271,29 @@ To lock the immutability policy:
        --if-match $etag
    ```
 
+### S3 Object Lock and working with snapshots
+
+#### Object Lock
+
+- S3 Object Lock blocks permanent object deletion for a user defined retention period.
+- It works on WORM(write once read many) model.
+- With S3 object lock, S3 versioning will automatically get enabled, it only prevent locked object versions from being permanently deleted.
+
+   > Note: The consumer of etcd-backup-restore must enable the object lock with the appropriate settings on their buckets to consume this feature. This is because backup-restore doesn't manage or interfere with the bucket (object store) creation process.
+
+#### Working with snapshots
+
+- S3 Object Lock can be activated at either on the bucket or object level. Moreover, it can be enabled when creating a new buckets or on a existing buckets.
+- For new buckets: These buckets will only contains the new snapshots, hence all the snapshots inside this bucket will be versioned locked snapshots.
+- For existing/old buckets: These buckets can contain a mix of pre-existing non-versioned, non-locked snapshots and new snapshots which are versioned and locked with retention period.
+The following diagram illustrates the working of snapshots with S3 for existing/old buckets as well as for new buckets.
+
+  ![Working with S3](../images/S3_immutability_working.png)
+
 ---
 
+> Note: If immutable snapshots are not enabled then the object's immutability expiry will be considered as zero, hence causing no effect on current functionality.
+
 ## Ignoring Snapshots During Restoration
 
 In certain scenarios, you might want `etcd-backup-restore` to ignore specific snapshots present in the object store during the restoration of etcd's data directory. When snapshots were mutable, operators could simply delete these snapshots, and subsequent restorations would not include them. However, once immutability is enabled, it is no longer possible to delete these snapshots.
@@ -319,6 +371,14 @@ To add the tag:
 
 After adding the annotation or tag, `etcd-backup-restore` will ignore these snapshots during the restoration process.
 
+#### AWS S3
+
+  - This method of tagging the snapshots to skip any snapshots during restoration is not supported for `AWS S3` buckets.
+  - For object lock, S3 object versioning will automatically get enabled. So this extra handling is not required as user can simply soft delete those snapshots.
+  - With object versioning inplace, a deletion marker will get added on top of those snapshots, and during the restoration of backup-restore, it will only considers the latest snapshots.
+  - If you want your snapshot back, just delete the deletio-marker.
+  - For more info: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html
+
 ---
 
 ## Setting the Immutability Period
@@ -369,5 +429,7 @@ By following best practices and regularly reviewing your backup and immutability
   - [Configure Immutability Policies](https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-container-scope)
   - [Blob Index Tags](https://learn.microsoft.com/azure/storage/blobs/storage-index-tags-overview)
 
----
-
+- **AWS S3**
+  - [Object Lock Documentation](https://aws.amazon.com/s3/features/object-lock/)
+  - [Object Lock policies]()
+  - [Deletion of object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html)

docs/usage/immutable_snapshots.md

@@ -0,0 +1,136 @@
+# Immutable Snapshots
+
+## Overview of Immutable Objects
+
+Several cloud providers offer functionality to create immutable objects within their storage services. Once an object is uploaded, it cannot be modified or deleted for a set period, known as the **immutability period**. These are referred to as **immutable objects**.
+
+Currently, etcd-backup-restore supports the use of immutable objects on the following cloud platforms:
+
+- Google Cloud Storage
+- Azure Blob Storage
+- AWS S3
+
+## Enabling and using Immutable Snapshots with etcd-backup-restore
+
+Etcd-backup-restore supports immutable objects, typically at what cloud providers call the "bucket level." During the creation of a bucket, it is configured to render objects immutable for a specific duration from the moment of their upload. This feature can be enabled through:
+
+- **Google Cloud Storage**: [Bucket Lock](https://cloud.google.com/storage/docs/bucket-lock)
+- **Azure Blob Storage**: [Container-level WORM Policies](https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-container-level-worm-policies)
+- **AWS S3**: [S3 object Lock](https://aws.amazon.com/s3/features/object-lock/)
+
+It is also possible to enable immutability retroactively by making appropriate API calls to your cloud provider, allowing the immutable snapshots feature to be used with existing buckets. For information on such configurations, please refer to your cloud provider's documentation.
+
+Setting the immutability period at the bucket level through the CLI can be done through the following commands:
+
+- Google Cloud Storage would be done like so:
+
+```sh
+gcloud storage buckets update <your-bucket> --retention-period=<desired-immutability-period>
+```
+
+- Azure Blob Storage Container would be done like so:
+
+```sh
+az storage container immutability-policy create --resource-group <your-resource-group> --account-name <your-account-name> --container-name <your-container-name> --period <desired-immutability-period>
+```
+
+- AWS S3:
+
+  1. To enable the object lock on new buckets
+
+   * Create a new bucket with object lock then update the bucket with object lock configuration.
+
+```bash
+# create new bucket with object lock enabled
+aws s3api create-bucket --bucket <your-bucket-name> --region <region> --create-bucket-configuration LocationConstraint=<region> --object-lock-enabled-for-bucket
+
+# update the bucket with object lock configuration
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
+
+  2. To enable the object lock on existing buckets
+
+  * First enable the object versioning on existing bucket then enable the object lock on bucket with its configurations (say `X` retention period).
+
+```bash
+# enable the object versioning on existing bucket
+aws s3api put-bucket-versioning --bucket <your-bucket-name> --versioning-configuration Status=Enabled
+
+# now, enable the object lock on bucket with its configurations
+aws s3api put-object-lock-configuration --bucket <your-bucket-name> --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE/GOVERNANCE", "Days": X }}}'
+```
+
+The behaviour of bucket's objects uploaded before a bucket is set to immutable varies among storage providers. etcd-backup-restore manages these objects and will perform garbage collection according to the configured garbage collection policy and the object's immutability expiry.
+
+> Note: If immutable snapshots are not enabled then the object's immutability expiry will be considered as zero, hence causing no effect on current functionality.
+
+## Current Capabilities
+
+etcd-backup-restore does not require configurations related to the immutability period of bucket's objects as this information is derived from the bucket's existing immutability settings. The etcd-backup-restore process also verifies the immutability expiry time of an object prior to initiating its garbage collection.
+
+Therefore, it is advisable to configure your garbage collection policies based on the duration you want your objects to remain immutable.
+
+## Storage Considerations
+
+Making objects immutable for extended periods can increase storage costs since these objects cannot be removed once uploaded. Storing outdated snapshots beyond their utility does not significantly enhance recovery capabilities. Therefore, consider all factors before enabling immutability for buckets, as this feature is irreversible once set by cloud providers.
+
+## Ignoring Snapshots From Restoration
+
+There might be certain cases where operators would like `etcd-backup-restore` to ignore particular snapshots present in the object store during restoration of etcd's data-dir.
+When snapshots were mutable, operators could simply delete these snapshots, and the restoration that follows this would not include them.
+Once immutability is turned on, however, it would not be possible to do this.
+
+Various cloud providers provide functionality to add custom annotations/tags to objects to add additional information to objects. These additional annotations/tags are orthogonal to the object's metadata, and therefore do not affect the object itself.  This feature is thus available for objects which are immutable as well.
+
+We leverage this feature to signal to etcd-backup-restore to not consider certain snapshots during restoration.
+The annotation/tag that is to be added to a snapshot for this is `x-etcd-snapshot-exclude=true`.
+
+You can add these tags through for the following providers like so:
+
+- **Google Cloud Storage**: as specified in the [docs](https://cloud.google.com/sdk/gcloud/reference/storage/objects/update?hl=en). (GCS calls this Custom Metadata).
+
+    ```sh
+    gcloud storage objects update gs://bucket/your-snapshot --custom-metadata=x-etcd-snapshot-exclude=true
+    ```
+
+    or:
+
+    Use the Google Cloud Console to add custom metadata to the object in the `Custom metadata` section of the object.
+
+- **Azure Blob Storage**: as specified in the [docs](https://learn.microsoft.com/en-us/cli/azure/storage/blob/tag?view=azure-cli-latest#az-storage-blob-tag-set). (ABS calls this tags).
+
+    ```sh
+    az storage blob tag set --container-name your-container --name your-snapshot --tags "x-etcd-snapshot-exclude"="true"
+    ```
+
+    or
+
+    Use the Azure Portal to add the tag in the `Blob index tags` section of the blob.
+
+Once these annotations/tags are added, etcd-backup-restore will ignore those snapshots during restoration.
+
+- **AWS S3**:
+
+  - This method of tagging the snapshots to skip any snapshots during restoration is not supported for `AWS S3` buckets.
+  - For object lock, S3 object versioning will automatically get enabled. So this extra handling is not required as user can simply soft delete those snapshots.
+  - With object versioning inplace, a deletion marker will get added on top of those snapshots, and during the restoration of backup-restore, it will only considers the latest snapshots.
+  - For more info: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html
+
+### S3 Object Lock and working with snapshots
+
+#### Object Lock
+
+- S3 Object Lock blocks permanent object deletion for a user defined retention period.
+- It works on WORM(write once read many) model.
+- With S3 object lock, S3 versioning will automatically get enabled, it only prevent locked object versions from being permanently deleted.
+
+   > Note: The consumer of etcd-backup-restore must enable the object lock with the appropriate settings on their buckets to consume this feature. This is because backup-restore doesn't manage or interfere with the bucket (object store) creation process.
+
+#### Working with snapshots
+
+- S3 Object Lock can be activated at either on the bucket or object level. Moreover, it can be enabled when creating a new buckets or on a existing buckets.
+- For new buckets: These buckets will only contains the new snapshots, hence all the snapshots inside this bucket will be versioned locked snapshots.
+- For existing/old buckets: These buckets can contain a mix of pre-existing non-versioned, non-locked snapshots and new snapshots which are versioned and locked with retention period.
+The following diagram illustrates the working of snapshots with S3 for existing/old buckets as well as for new buckets.
+
+  ![Working with S3](../images/S3_immutability_working.png)

pkg/snapshot/snapshotter/garbagecollector.go

@@ -181,6 +181,10 @@ func (ssr *Snapshotter) RunGarbageCollector(stopCh <-chan struct{}) {
 					if fullSnapshotIndex < len(fullSnapshotIndexList)-int(ssr.config.MaxBackups) {
 						snap := snapList[fullSnapshotIndexList[fullSnapshotIndex]]
 						snapPath := path.Join(snap.SnapDir, snap.SnapName)
+						if !snap.IsDeletable() {
+							ssr.logger.Infof("GC: Skipping the snapshot: %s, since its immutability period hasn't expired yet", snap.SnapName)
+							continue
+						}
 						ssr.logger.Infof("GC: Deleting old full snapshot: %s", snapPath)
 						if err := ssr.store.Delete(*snap); errors.Is(err, brtypes.ErrSnapshotDeleteFailDueToImmutability) {
 							// The snapshot is still immutable, attempt to gargbage collect it in the next run

pkg/snapstore/oss_snapstore.go

@@ -52,7 +52,6 @@ type authOptions struct {
 type OSSSnapStore struct {
 	prefix                  string
 	bucket                  OSSBucket
-	multiPart               sync.Mutex
 	maxParallelChunkUploads uint
 	minChunkSize            int64
 	tempDir                 string