From 3d31b6f227f09391fb0f186ef879d1fb2ba33a75 Mon Sep 17 00:00:00 2001 From: Nils Hanke Date: Wed, 29 Mar 2023 11:17:54 +0200 Subject: [PATCH 1/4] dev-docs: add rough etcd disaster recovery guide --- dev-docs/howto/etcd-disaster-recovery.md | 139 +++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 dev-docs/howto/etcd-disaster-recovery.md diff --git a/dev-docs/howto/etcd-disaster-recovery.md b/dev-docs/howto/etcd-disaster-recovery.md new file mode 100644 index 0000000000..113ff602d3 --- /dev/null +++ b/dev-docs/howto/etcd-disaster-recovery.md @@ -0,0 +1,139 @@ +# etcd disaster recovery +When etcd loses quorum and (N-1)/2 control planes are lost, +etcd will not be able to perform any transactions anymore and will basically stall and cause timeouts. +This makes the Constellation control planes unusable, resulting in a frozen cluster. The worker nodes will continue to function for a bit, +but given that they cannot communicate to the control plane anymore, they will eventually also cease to function correctly. + +If the missing control plane nodes are still existent and their state disk still exists, you likely do not need this guide. +If the missing are irrecoverably lost (e.g. scaled up and down the control plane instance set), follow through this guide to bring the cluster back up. + +## General concept +1. Create snapshot of a state disk from a remaining control plane node. +2. Download the snapshot and decrypt it locally +3. Follow the [restoring a cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) guide from etcd. +4. Save the modified virtual disk and upload it back to the CSP. +5. Modify the scale set (or remaining VM singularly, if you can) to use the new uploaded data disk. +6. Reboot, wait a few minutes. +7. Pray it workded ;) + +## How I did it once (Azure) + +1. If the VM has never been rebooted once after initialization, reboot it once to sync any LUKS passphrase changes to disk (not 100% sure if necessary to sync the change to the passphrase - would have to double-check that later with an experimental cluster) + +2. Create a snapshot from the disk using the CLI: +```bash +az snapshot create --resource-group dogfooding --name dogfooding-3 --source /subscriptions/0d202bbb-4fa7-4af8-8125-58c269a05435/resourceGroups/dogfooding/providers/Microsoft.Compute/disks/constell-f2332c74-coconstell-f2332c74-condisk2_dd460a6ae3124aa3a4c23be0ab39634e --location northeurope +``` + +3. Look up the snapshot online, export it as VHD +Mount the disk: +```bash +modprobe nbd && sudo qemu-nbd -c /dev/nbd0 /home/nils/Downloads/constellation-disk.vhd +``` + +4. Get the UUID of the disk: +```bash +sudo cryptsetup luksDump /dev/nbd0 +``` + +5. Regenerate the passphrase to unlock the disk (the [code snippet below](#get-disk-decryption-key) might be useful for this) + +6. Decrypt the disk: +```bash +sudo cryptsetup luksOpen /dev/nbd0 constellation-state --key-file passphrase +``` + +7. Mount the decrypted disk (I just did this via the Nautilus) + +8. Find the db file from etcd in `/var/lib/etcd/member/snap/db` + +9. Perform the etcd [Restoring a Cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) step: + +```bash +./etcdutl snapshot restore db --initial-cluster constell-f2332c74-control-plane000001=https://10.9.126.0:2380 --initial-advertise-peer-urls https://10.9.126.0:2380 --data-dir recovery --name constell-f2332c74-control-plane000001 --skip-hash-check=true +``` +*(replace name & IP with the name & the private IP of the remaining control plane VM - this can be seen in the Azure portal)* + +10. Copy the contents of the newly created recovery directory to the mounted state disk and remove any remaining old files. +**Make sure the permissions are the same as before!** + +11. Unmount the partition: +```bash +sudo umount /your/mount/path +sudo luksClose constellation-state +sudo qemu-nbd -d /dev/nbd0 +``` + +12. Upload the modified VHD back to Azure (I just used Azure Storage Explorer for this). + +13. Patch the whole control-plane VMSS to remove LUN 0 from the VMs: +```bash +az vmss disk detach --lun 0 --resource-group dogfooding --vmss-name constell-f2332c74-control-plane +``` + +14. Update the VM: +```bash +az vmss update-instances -g dogfooding --name constell-f2332c74-control-plane --instance-ids 1 +``` + +15. Attach the uploaded disk as LUN 0 (either via CLI or Azure Portal, I just used the Azure Portal). + +16. Start the VM and pray it works ;) It can take a few minutes before etcd becomes fully alive again. + +17. Patch the state disk definition back to the VMSS (no idea how, haven't done his yet) so newly created VMs in the VMSS have a clean state disk again. + +## Get disk decryption key +```golang +package main + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "os" + + "golang.org/x/crypto/hkdf" +) + +type MasterSecret struct { + Key []byte `json:"key"` + Salt []byte `json:"salt"` +} + +func main() { + uuid := "4ae66293-57aa-4821-b99c-ebc598a6e5a8" // replace me + + masterSecretRaw, err := os.ReadFile("constellation-mastersecret.json") + if err != nil { + panic(err) + } + + var masterSecret MasterSecret + if err := json.Unmarshal(masterSecretRaw, &masterSecret); err != nil { + panic(err) + } + + dek, err := DeriveKey(masterSecret.Key, masterSecret.Salt, []byte("key-"+uuid), 32) + if err != nil { + panic(err) + } + + fmt.Println(hex.EncodeToString(dek)) + + if err := os.WriteFile("passphrase", dek, 0o644); err != nil { + panic(err) + } +} + +// DeriveKey derives a key from a secret. +func DeriveKey(secret, salt, info []byte, length uint) ([]byte, error) { + hkdfReader := hkdf.New(sha256.New, secret, salt, info) + key := make([]byte, length) + if _, err := io.ReadFull(hkdfReader, key); err != nil { + return nil, err + } + return key, nil +} +``` From 6612b5cf9b730688a4a7c81df44e1150e947eebb Mon Sep 17 00:00:00 2001 From: Nils Hanke Date: Wed, 29 Mar 2023 11:21:16 +0200 Subject: [PATCH 2/4] dev-docs: use same spelling from etcd guide --- dev-docs/howto/etcd-disaster-recovery.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dev-docs/howto/etcd-disaster-recovery.md b/dev-docs/howto/etcd-disaster-recovery.md index 113ff602d3..b18956c06d 100644 --- a/dev-docs/howto/etcd-disaster-recovery.md +++ b/dev-docs/howto/etcd-disaster-recovery.md @@ -10,7 +10,7 @@ If the missing are irrecoverably lost (e.g. scaled up and down the control plane ## General concept 1. Create snapshot of a state disk from a remaining control plane node. 2. Download the snapshot and decrypt it locally -3. Follow the [restoring a cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) guide from etcd. +3. Follow the [Restoring a cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) guide from etcd. 4. Save the modified virtual disk and upload it back to the CSP. 5. Modify the scale set (or remaining VM singularly, if you can) to use the new uploaded data disk. 6. Reboot, wait a few minutes. @@ -47,7 +47,7 @@ sudo cryptsetup luksOpen /dev/nbd0 constellation-state --key-file passphrase 8. Find the db file from etcd in `/var/lib/etcd/member/snap/db` -9. Perform the etcd [Restoring a Cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) step: +9. Perform the etcd [Restoring a cluster](https://etcd.io/docs/v3.5/op-guide/recovery/#restoring-a-cluster) step: ```bash ./etcdutl snapshot restore db --initial-cluster constell-f2332c74-control-plane000001=https://10.9.126.0:2380 --initial-advertise-peer-urls https://10.9.126.0:2380 --data-dir recovery --name constell-f2332c74-control-plane000001 --skip-hash-check=true From 13db77d06e6b8a8cd830afa3cf1a7f4e61757ee4 Mon Sep 17 00:00:00 2001 From: Nils Hanke Date: Wed, 29 Mar 2023 11:21:26 +0200 Subject: [PATCH 3/4] dev-docs: slight rewording --- dev-docs/howto/etcd-disaster-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-docs/howto/etcd-disaster-recovery.md b/dev-docs/howto/etcd-disaster-recovery.md index b18956c06d..80465d0a9c 100644 --- a/dev-docs/howto/etcd-disaster-recovery.md +++ b/dev-docs/howto/etcd-disaster-recovery.md @@ -52,7 +52,7 @@ sudo cryptsetup luksOpen /dev/nbd0 constellation-state --key-file passphrase ```bash ./etcdutl snapshot restore db --initial-cluster constell-f2332c74-control-plane000001=https://10.9.126.0:2380 --initial-advertise-peer-urls https://10.9.126.0:2380 --data-dir recovery --name constell-f2332c74-control-plane000001 --skip-hash-check=true ``` -*(replace name & IP with the name & the private IP of the remaining control plane VM - this can be seen in the Azure portal)* +*(replace name & IP with the name and the private IP of the remaining control plane VM you are to perform the restore process on - this information can be found in the Azure portal)* 10. Copy the contents of the newly created recovery directory to the mounted state disk and remove any remaining old files. **Make sure the permissions are the same as before!** From d7314f5e754e7bf63fc843e8fcb7075c845b860c Mon Sep 17 00:00:00 2001 From: Nils Hanke Date: Wed, 29 Mar 2023 13:58:42 +0200 Subject: [PATCH 4/4] fix typo --- dev-docs/howto/etcd-disaster-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-docs/howto/etcd-disaster-recovery.md b/dev-docs/howto/etcd-disaster-recovery.md index 80465d0a9c..3d180a9058 100644 --- a/dev-docs/howto/etcd-disaster-recovery.md +++ b/dev-docs/howto/etcd-disaster-recovery.md @@ -14,7 +14,7 @@ If the missing are irrecoverably lost (e.g. scaled up and down the control plane 4. Save the modified virtual disk and upload it back to the CSP. 5. Modify the scale set (or remaining VM singularly, if you can) to use the new uploaded data disk. 6. Reboot, wait a few minutes. -7. Pray it workded ;) +7. Pray it worked ;) ## How I did it once (Azure)