Skip to content

Commit

Permalink
updates
Browse files Browse the repository at this point in the history
  • Loading branch information
MarcoIeni committed Aug 20, 2024
1 parent a1657f1 commit c5679a4
Showing 1 changed file with 12 additions and 5 deletions.
17 changes: 12 additions & 5 deletions service-catalog/gcp-backup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,16 @@ The accidental account deletion is not a threat anymore because if either AWS or
## Implementation details

The account where we store the backup is called `rust-backup`. It contains two GCP projects: `backup-prod` and `backup-staging`.
Here we have one Google [Object Storage](https://cloud.google.com/storage?hl=en) in the `europe-west-1` (Belgium) region for the following AWS S3 buckets:
Here we have one Google [Object Storage](https://cloud.google.com/storage?hl=en) in the `europe-west1` (Belgium) region for the following AWS S3 buckets:

- `crates-io`. Cloudfront URL: `cloudfront-static.crates.io`. It contains the crates published by the Rust community.
- `static-rust-lang-org`. Cloudfront Url: `cloudfront-static.rust-lang.org`. Among other things, it contains the Rust releases.

The [storage class](https://cloud.google.com/storage/docs/storage-classes) is set to "archive" for both buckets. This is the cheapest class for infrequent access.
For the objects:
- Set the [storage class](https://cloud.google.com/storage/docs/storage-classes) to "archive" for all buckets.
This is the cheapest class for infrequent access.
- Enable [object-versioning](https://cloud.google.com/storage/docs/object-versioning) and [soft-delete](https://cloud.google.com/storage/docs/soft-delete),
so that we can recover updates and deletes so that we can recover updates and deletes.

We use [Storage Transfer](https://cloud.google.com/storage-transfer/docs/overview) to automatically transfer the content of the s3 bucket into the Google Object Storage.
This is a service managed by Google. We'll use it to download the S3 buckets from cloudfront to perform a daily incremental transfer. The transfers only move files that are new, updated, or deleted since the last transfer, minimizing the amount of data that needs to be transferred.
Expand All @@ -96,17 +100,20 @@ You can also run the following test:
- Edit the file in AWS and check that you can recover the previous version from GCP.
- Delete the in AWS and check that you can recover all previous versions from GCP.

In the future, we might want to create alarts in Datadog to monitor if the transfer job fails.
In the future, we might want to create alerts in:
- _Datadog_: to monitor if the transfer job fails.
- _Wiz_: to monitor if the access control changes.

### FAQ 🤔

#### Do we need a multi-region backup for the object storage?

No. [Multi-region](https://cloud.google.com/storage/docs/availability-durability#cross-region-redundancy) only helps if we want to serve this data real-time and we want to have a fallback mechanism if a GCP region fails. We just need this object storage for backup purposes, so we don't need to pay more 👍

#### Why did you choose the `us-west-1` GCP region?
#### Why did you choose the `europe-west1` GCP region?

It's the same region where the AWS S3 buckets are located. In this way, we reduce the latency of the transfer job.
It's far from the `us-west-1` region where the AWS S3 buckets are located. This protects us from geographical disasters.
The con is that the latency of the transfer job is higher when compared to a region in the US.
Also, the cost calculator indicates that this regions has a "Low CO2" and it's among the cheapest regions.

#### Why GCP?
Expand Down

0 comments on commit c5679a4

Please sign in to comment.