diff --git a/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg b/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg new file mode 100644 index 00000000..2568b46d --- /dev/null +++ b/docs/modules/ROOT/assets/images/emergency-controller-credentials.svg @@ -0,0 +1,4 @@ + + + +
Cluster A
Cluster A
Cluster B
Cluster B
Cluster C
Cluster C
commodore managed
manifests
commodore managed...
git.vshn.net
global defaults
git.vshn.net...
retrieve publickeys
retrieve publickeys
Passbolt
Passbolt
S3
S3
public key
encrypted secrets
public key...
git commit && push
git commit && push
sync pipeline
sync pipeline
retrieval tool
retrieval tool
OnCall
engineer
OnCal...
private key
private...
public keys
public k...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc new file mode 100644 index 00000000..ed34ff55 --- /dev/null +++ b/docs/modules/ROOT/pages/references/architecture/emergency_credentials.adoc @@ -0,0 +1,128 @@ += Cluster Emergency Credentials + +== Problem Statement + +We started using our https://github.com/appuio/emergency-credentials-controller[emergency credentials controller] to rotate emergency credentials for clusters. + +We need a way store and retrieve those emergency credentials for clusters. +We should be able to retrieve the credentials in an emergency situation where we lost access to our internal systems. + +VSHN uses https://www.passbolt.com/[passbolt] to store secrets independently of our internal systems. +Passbolt uses asymetric encryption to store secrets. + +In order to share secrets using passbolt you need to be in the possession of a private key that can decrypt the secret. +Since passbolt login and decryption keys can't be separated, or rotated, at the moment, we would need to protect the private keys very carefully. + +== High Level Goals + +* Clusters should be able to write their emergency credentials automatically and securely +* Clusters must not be able to read other clusters emergency credentials +* Clusters shouldn't be able to read other clusters metadata +* VSHNeers should be able to read the emergency credentials from passbolt without any VSHN infrastructure +* VSHNeers leaving the company shouldn't be able to retain access to the (refreshed) emergency credentials + +== Non-Goals + +* Storing secrets independent of infrastructure + +== Implementation + +image:emergency-controller-credentials.svg[] + +The emergency credentials controller gets a new S3 integration that allows encrypting the credentials with a list of public keys and then pushing them to S3. + +A pipeline in the VSHN Gitlab will export public keys for all users in the On-Call group and check them into the commodore defaults repository. + +A utility tool is provided to download the encrypted credentials for a cluster from S3 and decrypt them using the passbolt private key. + +=== Emergency Credentials Controller + +The controller can be configured to use a new S3 integration that allows encrypting the credentials with a list of public keys and then pushing them to S3. + +[source,yaml] +---- +tokenStores: + - name: s3 + type: s3 + s3: + pathTemplate: "{{ env CLUSTER_ID | sha256sum }}-{{ now | unixEpoch }}.gpg" <1> + endpoint: + bucketnames: bucket_name1, bucket_name2 + endpoint: s3.endpoint.com + region: s3_region + access_key_id: s3_access_key_id + secret_access_key: s3_secret_access_key + http_config: {} + s3forcepathstyle: true + encryption: + type: gpg + publicKeys: [] <2> + publicKeysFile: "" <3> +---- +<1> https://masterminds.github.io/sprig/[Sprig template] that will be used to generate the path for the uploaded credentials file. +The cluster ID here is sha256 hashed to prevent leaking metadata about the clusters if using a shared bucket for multiple clusters. +<2> List will be concatenated +<3> A concatenated list of public keys will be read from this file + +==== Uploaded file format + +The uploaded file will be a JSON file with the following format: + +[source,json] +---- +{ + "secrets": [ + { + "data": "-----BEGIN PGP MESSAGE-----\nVersion: GopenPGP 2.7.4\nComment: https://gopenpgp.org\n\n[...]\n-----END PGP MESSAGE-----", + }, + { + "data": "-----BEGIN PGP MESSAGE-----\nVersion: GopenPGP 2.7.4\nComment: https://gopenpgp.org\n\n[...]\n-----END PGP MESSAGE-----", + } + ] +} +---- + +The `data` field contains the encrypted secret. +A client should try to decrypt the secret with all available private keys until one succeeds. + +=== Public Key Sync + +A scheduled pipeline in the VSHN Gitlab will export public keys for all users in the On-Call group and check them into the commodore defaults repository. + +The pipeline has access to a passbolt User without any access to the secrets. +Every passbolt user is allowed to export every other users public key. +It will export the public keys for all users in the On-Call group and check them into the commodore defaults repository. + +We currently compile all cluster repositories daily, so the public keys will be updated daily. + +=== Utility Tool + +A utility tool is provided to download the encrypted credentials for a cluster from S3 and decrypt them using the passbolt private key. + +The tool receives the secret location and any bucket access keys from passbolt. +The tool receives the cluster ID as a parameter and hashes/formats it to match the path template used by the emergency credentials controller. +It then downloads the file from S3 and tries to decrypt it using the passbolt private key. +After decryption it prints the secrets to stdout with an optional kubeconfig file for the cluster. + +[source,bash] +---- +$ cat > ~/.retrieve-emergency-credentials/config.yaml << YAML +passbolt: + url: "https://passbolt.vshn.net" + key: | + ASCII ARMORED PUBLIC KEY + privateKey: | + ASCII ARMORED PRIVATE KEY +YAML + +$ retrieve-emergency-credentials cluster-id +Token: [...] +Kubeconfig exported to /tmp/kubeconfig-XXXXXX. You can access the cluster with: + export KUBECONFIG=/tmp/kubeconfig-XXXXXX + kubectl cluster-info +---- + +== Resources + +- https://pkg.go.dev/github.com/passbolt/go-passbolt@v0.7.0/api#Client.GetUsers[`passbolt-go.Client.GetUsers`] +- https://github.com/ProtonMail/gopenpgp[ProtonMails Go OpenPGP library] diff --git a/docs/modules/ROOT/partials/nav.adoc b/docs/modules/ROOT/partials/nav.adoc index 87b19e7c..1e3adfb2 100644 --- a/docs/modules/ROOT/partials/nav.adoc +++ b/docs/modules/ROOT/partials/nav.adoc @@ -10,6 +10,7 @@ ** xref:oc4:ROOT:explanations/etcd_encryption.adoc[] ** xref:oc4:ROOT:explanations/pod_security.adoc[] ** xref:oc4:ROOT:references/architecture/upgrade_controller.adoc[Upgrade Controller] +** xref:oc4:ROOT:references/architecture/emergency_credentials.adoc[] ** xref:oc4:ROOT:references/architecture/metering-data-flow-appuio-managed.adoc[Resource Usage Reporting] ** xref:oc4:ROOT:references/architecture/single_sign_on.adoc[] @@ -97,6 +98,7 @@ ** xref:oc4:ROOT:how-tos/update_maintenance/v_4_12.adoc[Upgrade to OCP4.12] ** xref:oc4:ROOT:how-tos/update_maintenance/v_4_13.adoc[Upgrade to OCP4.13] ** xref:oc4:ROOT:references/architecture/upgrade_controller.adoc[Upgrade Controller] +** xref:oc4:ROOT:references/architecture/emergency_credentials.adoc[] ** xref:oc4:ROOT:how-tos/maintenance_troubleshooting.adoc[Maintenance troubleshooting] // Support