From c5679a42efcd6bb696c8854b6c5cf0d06ed6385a Mon Sep 17 00:00:00 2001 From: MarcoIeni <11428655+MarcoIeni@users.noreply.github.com> Date: Tue, 20 Aug 2024 11:55:35 +0200 Subject: [PATCH] updates --- service-catalog/gcp-backup/README.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/service-catalog/gcp-backup/README.md b/service-catalog/gcp-backup/README.md index 2956dde..443bc62 100644 --- a/service-catalog/gcp-backup/README.md +++ b/service-catalog/gcp-backup/README.md @@ -73,12 +73,16 @@ The accidental account deletion is not a threat anymore because if either AWS or ## Implementation details The account where we store the backup is called `rust-backup`. It contains two GCP projects: `backup-prod` and `backup-staging`. -Here we have one Google [Object Storage](https://cloud.google.com/storage?hl=en) in the `europe-west-1` (Belgium) region for the following AWS S3 buckets: +Here we have one Google [Object Storage](https://cloud.google.com/storage?hl=en) in the `europe-west1` (Belgium) region for the following AWS S3 buckets: - `crates-io`. Cloudfront URL: `cloudfront-static.crates.io`. It contains the crates published by the Rust community. - `static-rust-lang-org`. Cloudfront Url: `cloudfront-static.rust-lang.org`. Among other things, it contains the Rust releases. -The [storage class](https://cloud.google.com/storage/docs/storage-classes) is set to "archive" for both buckets. This is the cheapest class for infrequent access. +For the objects: +- Set the [storage class](https://cloud.google.com/storage/docs/storage-classes) to "archive" for all buckets. + This is the cheapest class for infrequent access. +- Enable [object-versioning](https://cloud.google.com/storage/docs/object-versioning) and [soft-delete](https://cloud.google.com/storage/docs/soft-delete), + so that we can recover updates and deletes so that we can recover updates and deletes. We use [Storage Transfer](https://cloud.google.com/storage-transfer/docs/overview) to automatically transfer the content of the s3 bucket into the Google Object Storage. This is a service managed by Google. We'll use it to download the S3 buckets from cloudfront to perform a daily incremental transfer. The transfers only move files that are new, updated, or deleted since the last transfer, minimizing the amount of data that needs to be transferred. @@ -96,7 +100,9 @@ You can also run the following test: - Edit the file in AWS and check that you can recover the previous version from GCP. - Delete the in AWS and check that you can recover all previous versions from GCP. -In the future, we might want to create alarts in Datadog to monitor if the transfer job fails. +In the future, we might want to create alerts in: +- _Datadog_: to monitor if the transfer job fails. +- _Wiz_: to monitor if the access control changes. ### FAQ 🤔 @@ -104,9 +110,10 @@ In the future, we might want to create alarts in Datadog to monitor if the trans No. [Multi-region](https://cloud.google.com/storage/docs/availability-durability#cross-region-redundancy) only helps if we want to serve this data real-time and we want to have a fallback mechanism if a GCP region fails. We just need this object storage for backup purposes, so we don't need to pay more 👍 -#### Why did you choose the `us-west-1` GCP region? +#### Why did you choose the `europe-west1` GCP region? -It's the same region where the AWS S3 buckets are located. In this way, we reduce the latency of the transfer job. +It's far from the `us-west-1` region where the AWS S3 buckets are located. This protects us from geographical disasters. +The con is that the latency of the transfer job is higher when compared to a region in the US. Also, the cost calculator indicates that this regions has a "Low CO2" and it's among the cheapest regions. #### Why GCP?