From 9f6a0e43702dae114f4d76d53e10a84339f9c168 Mon Sep 17 00:00:00 2001 From: Sebastian Widmer Date: Fri, 15 Dec 2023 17:28:50 +0100 Subject: [PATCH 1/2] Emergency credentials controller docs --- .../emergency-controller-credentials.svg | 4 + .../architecture/emergency_credentials.adoc | 128 ++++++++++++++++++ docs/modules/ROOT/partials/nav.adoc | 2 + 3 files changed, 134 insertions(+) create mode 100644 docs/modules/ROOT/assets/images/emergency-controller-credentials.svg create mode 100644 docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc diff --git a/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg b/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg new file mode 100644 index 00000000..2568b46d --- /dev/null +++ b/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg @@ -0,0 +1,4 @@ + + + +
Cluster A
Cluster A
Cluster B
Cluster B
Cluster C
Cluster C
commodore managed
manifests
commodore managed...
git.vshn.net
global defaults
git.vshn.net...
retrieve publickeys
retrieve publickeys
Passbolt
Passbolt
S3
S3
public key
encrypted secrets
public key...
git commit && push
git commit && push
sync pipeline
sync pipeline
retrieval tool
retrieval tool
OnCall
engineer
OnCal...
private key
private...
public keys
public k...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc new file mode 100644 index 00000000..95a37cf8 --- /dev/null +++ b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc @@ -0,0 +1,128 @@ += Cluster Emergency Credentials + +== Problem Statement + +We started using our https://github.com/appuio/emergency-credentials-controller[emergency credentials controller] to rotate emergency credentials for clusters. + +We need a way store and retrieve those emergency credentials for clusters. +We should be able to retrieve the credentials in an emergency situation where we lost access to our internal systems. + +VSHN uses https://www.passbolt.com/[passbolt] to store secrets independently of our internal systems. +Passbolt uses asymetric encryption to store secrets. + +In order to share secrets using passbolt you need to be in the possession of a private key that can decrypt the secret. +Since passbolt login and decryption keys can't be separated, or rotated, at the moment, we would need to protect the private keys very carefully. + +== High Level Goals + +* Clusters should be able to write their emergency credentials automatically and securely +* Clusters must not be able to read other clusters emergency credentials +* Clusters shouldn't be able to read other clusters metadata +* VSHNeers should be able to read the emergency credentials from passbolt without any VSHN infrastructure +* VSHNeers leaving the company shouldn't be able to retain access to the (refreshed) emergency credentials + +== Non-Goals + +* Storing secrets independent of infrastructure + +== Implementation + +image:emergency-controller-credentials.svg[] + +The emergency credentials controller gets a new S3 integration that allows encrypting the credentials with a list of public keys and then pushing them to S3. + +A pipeline in the VSHN Gitlab will export public keys for all users in the On-Call group and check them into the commodore defaults repository. + +A utility tool is provided to download the encrypted credentials for a cluster from S3 and decrypt them using the passbolt private key. + +=== Emergency Credentials Controller + +The controller can be configured to use a new S3 integration that allows encrypting the credentials with a list of public keys and then pushing them to S3. + +[source,yaml] +---- +tokenStores: + - name: s3 + type: s3 + s3: + pathTemplate: "{{ env CLUSTER_ID | sha256sum }}-{{ now | unixEpoch }}.gpg" <1> + endpoint: + bucketnames: bucket_name1, bucket_name2 + endpoint: s3.endpoint.com + region: s3_region + access_key_id: s3_access_key_id + secret_access_key: s3_secret_access_key + http_config: {} + s3forcepathstyle: true + encryption: + type: gpg + publicKeys: [] <2> + publicKeysFile: "" <3> +---- +<1> https://masterminds.github.io/sprig/[Sprig template] that will be used to generate the path for the uploaded credentials file. +The cluster ID here is sha256 hashed to prevent leaking metadata about the clusters if using a shared bucket for multiple clusters. +<2> List will be concatenated +<3> A concatenated list of public keys will be read from this file + +==== Uploaded file format + +The uploaded file will be a JSON file with the following format: + +[source,json] +---- +{ + "secrets": [ + { + "data": "-----BEGIN PGP MESSAGE-----\nVersion: GopenPGP 2.7.4\nComment: https://gopenpgp.org\n\n[...]\n-----END PGP MESSAGE-----", + }, + { + "data": "-----BEGIN PGP MESSAGE-----\nVersion: GopenPGP 2.7.4\nComment: https://gopenpgp.org\n\n[...]\n-----END PGP MESSAGE-----", + } + ] +} +---- + +The `data` field contains the encrypted secret. +A client should try to decrypt the secret with all available private keys until one succeeds. + +=== Public Key Sync + +A scheduled pipeline in the VSHN Gitlab will export public keys for all users in the On-Call group and check them into the commodore defaults repository. + +The pipeline has access to a passbolt User without any access to the secrets. +Every passbolt user is allowed to export every other users public key. +It will export the public keys for all users in the On-Call group and check them into the commodore defaults repository. + +We currently compile all cluster repositories daily, so the public keys will be updated daily. + +=== Utility Tool + +A utility tool is provided to download the encrypted credentials for a cluster from S3 and decrypt them using the passbolt private key. + +The tool receives the secret location and any bucket access keys from passbolt. +The tool receives the cluster ID as a parameter and hashes/formats it to match the path template used by the emergency credentials controller. +It then downloads the file from S3 and tries to decrypt it using the passbolt private key. +After decryption it prints the secrets to stdout with an optional kubeconfig file for the cluster. + +[source,bash] +--- +$ cat > ~/.retrieve-emergency-credentials/config.yaml << YAML +passbolt: + url: "https://passbolt.vshn.net" + key: | + ASCII ARMORED PUBLIC KEY + privateKey: | + ASCII ARMORED PRIVATE KEY +YAML + +$ retrieve-emergency-credentials cluster-id +Token: [...] +Kubeconfig exported to /tmp/kubeconfig-XXXXXX. You can access the cluster with: + export KUBECONFIG=/tmp/kubeconfig-XXXXXX + kubectl cluster-info +--- + +== Resources + +- https://pkg.go.dev/github.com/passbolt/go-passbolt@v0.7.0/api#Client.GetUsers[`passbolt-go.Client.GetUsers`] +- https://github.com/ProtonMail/gopenpgp[ProtonMails Go OpenPGP library] diff --git a/docs/modules/ROOT/partials/nav.adoc b/docs/modules/ROOT/partials/nav.adoc index 87b19e7c..1e3adfb2 100644 --- a/docs/modules/ROOT/partials/nav.adoc +++ b/docs/modules/ROOT/partials/nav.adoc @@ -10,6 +10,7 @@ ** xref:oc4:ROOT:explanations/etcd_encryption.adoc[] ** xref:oc4:ROOT:explanations/pod_security.adoc[] ** xref:oc4:ROOT:references/architecture/upgrade_controller.adoc[Upgrade Controller] +** xref:oc4:ROOT:references/architecture/emergency_credentials.adoc[] ** xref:oc4:ROOT:references/architecture/metering-data-flow-appuio-managed.adoc[Resource Usage Reporting] ** xref:oc4:ROOT:references/architecture/single_sign_on.adoc[] @@ -97,6 +98,7 @@ ** xref:oc4:ROOT:how-tos/update_maintenance/v_4_12.adoc[Upgrade to OCP4.12] ** xref:oc4:ROOT:how-tos/update_maintenance/v_4_13.adoc[Upgrade to OCP4.13] ** xref:oc4:ROOT:references/architecture/upgrade_controller.adoc[Upgrade Controller] +** xref:oc4:ROOT:references/architecture/emergency_credentials.adoc[] ** xref:oc4:ROOT:how-tos/maintenance_troubleshooting.adoc[Maintenance troubleshooting] // Support From 310fc33ddaa42c676e6419df91700934a2dd47b5 Mon Sep 17 00:00:00 2001 From: Stephan Feurer Date: Mon, 18 Dec 2023 11:50:49 +0100 Subject: [PATCH 2/2] Fix codeblock --- .../pages/references/architecture/emergency_credentials.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc index 95a37cf8..ed34ff55 100644 --- a/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc +++ b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc @@ -105,7 +105,7 @@ It then downloads the file from S3 and tries to decrypt it using the passbolt pr After decryption it prints the secrets to stdout with an optional kubeconfig file for the cluster. [source,bash] ---- +---- $ cat > ~/.retrieve-emergency-credentials/config.yaml << YAML passbolt: url: "https://passbolt.vshn.net" @@ -120,7 +120,7 @@ Token: [...] Kubeconfig exported to /tmp/kubeconfig-XXXXXX. You can access the cluster with: export KUBECONFIG=/tmp/kubeconfig-XXXXXX kubectl cluster-info ---- +---- == Resources