diff --git a/docs/guides/mongodb/backup/kubestash/_index.md b/docs/guides/mongodb/backup/kubestash/_index.md index 3c326448f2..994241ef4f 100644 --- a/docs/guides/mongodb/backup/kubestash/_index.md +++ b/docs/guides/mongodb/backup/kubestash/_index.md @@ -5,6 +5,6 @@ menu: identifier: guides-mongodb-backup-kubestash name: KubeStash Backup & Restore parent: guides-mongodb-backup - weight: 20 + weight: 50 menu_name: docs_{{ .version }} ---- +--- \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/auto-backup/_index.md b/docs/guides/mongodb/backup/kubestash/auto-backup/_index.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/guides/mongodb/backup/kubestash/logical/_index.md b/docs/guides/mongodb/backup/kubestash/logical/_index.md new file mode 100644 index 0000000000..1aeabf00c9 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/_index.md @@ -0,0 +1,10 @@ +--- +title: Logical Backup & Restore of MongoDB | KubeStash +menu: + docs_{{ .version }}: + identifier: guides-mongodb-backup-kubestash-logical + name: Logical Backup & Restore + parent: guides-mongodb-backup-kubestash + weight: 20 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/_index.md b/docs/guides/mongodb/backup/kubestash/logical/replicaset/_index.md new file mode 100644 index 0000000000..79e990f3ec --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/_index.md @@ -0,0 +1,593 @@ +--- +title: Backup & Restore Replicaset MongoDB | KubeStash +description: Backup and restore replicaset MongoDB database using KubeStash +menu: + docs_{{ .version }}: + identifier: guides-mongodb-backup-kubestash-logical-replicaset + name: MongoDB Replicaset Cluster + parent: guides-mongodb-backup-kubestash-logical + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Backup and Restore MongoDB database using KubeStash + +KubeStash v0.1.0+ supports backup and restoration of MongoDB databases. This guide will show you how you can backup and restore your MongoDB database with KubeStash. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. +- Install KubeDB in your cluster following the steps [here](/docs/setup/README.md). +- Install KubeStash Enterprise in your cluster following the steps [here(link needed)](). +- Install KubeStash `kubectl` plugin following the steps [here(link needed)](). +- If you are not familiar with how KubeStash backup and restore MongoDB databases, please check the following guide [here](/docs/guides/mongodb/backup/kubestash/overview/index.md). + +You have to be familiar with following custom resources: + +- [AppBinding](/docs/guides/mongodb/concepts/appbinding.md) +- [Function](https://stash.run/docs/latest/concepts/crds/function/) +- [Task](https://stash.run/docs/latest/concepts/crds/task/) +- [BackupConfiguration](https://stash.run/docs/latest/concepts/crds/backupconfiguration/) +- [RestoreSession](https://stash.run/docs/latest/concepts/crds/restoresession/) + +To keep things isolated, we are going to use a separate namespace called `demo` throughout this tutorial. Create `demo` namespace if you haven't created yet. + +```console +$ kubectl create ns demo +namespace/demo created +``` + +## Backup MongoDB + +This section will demonstrate how to backup MongoDB database. Here, we are going to deploy a MongoDB database using KubeDB. Then, we are going to backup this database into a S3 bucket. Finally, we are going to restore the backed up data into another MongoDB database. + +### Deploy Sample MongoDB Database + +Let's deploy a sample MongoDB database and insert some data into it. + +**Create MongoDB CRD:** + +Below is the YAML of a sample MongoDB crd that we are going to create for this tutorial: + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-rs + namespace: demo +spec: + version: "4.2.3" + replicas: 3 + replicaSet: + name: rs0 + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset.yaml +mongodb.kubedb.com/sample-mg-rs created +``` + +KubeDB will deploy a MongoDB database according to the above specification. It will also create the necessary secrets and services to access the database. + +Let's check if the database is ready to use, + +```console +$ kubectl get mongodb -n demo sample-mg-rs +NAME VERSION STATUS AGE +sample-mg-rs 4.2.3 Ready 2m27s +``` + +The database is `Ready`. Verify that KubeDB has created a Secret and a Service for this database using the following commands, + +```console +$ kubectl get secret -n demo -l=app.kubernetes.io/instance=sample-mg-rs +NAME TYPE DATA AGE +sample-mg-rs-auth kubernetes.io/basic-auth 2 3m53s +sample-mg-rs-key Opaque 1 3m53s + +$ kubectl get service -n demo -l=app.kubernetes.io/instance=sample-mg-rs +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +sample-mg-rs ClusterIP 10.96.211.27 27017/TCP 4m38s +sample-mg-rs-pods ClusterIP None 27017/TCP 4m38s +``` + +Here, we have to use service `sample-mg-rs` and secret `sample-mg-rs-auth` to connect with the database. + +**Insert Sample Data:** + +> Note: You can insert data into this `MongoDB` database using our [KubeDB CLI](https://kubedb.com/docs/latest/setup/install/kubectl_plugin/). + +For simplicity, we are going to exec into the database pod and create some sample data. At first, find out the database pod using the following command, + +```console +$ kubectl get pods -n demo --selector="app.kubernetes.io/instance=sample-mg-rs" +NAME READY STATUS RESTARTS AGE +sample-mg-rs-0 2/2 Running 0 6m15s +sample-mg-rs-1 2/2 Running 0 5m36s +sample-mg-rs-2 2/2 Running 0 5m14s +``` + +Now, let's exec into the pod and create a table, + +```console +$ export USER=$(kubectl get secrets -n demo sample-mg-rs-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo sample-mg-rs-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo sample-mg-rs-0 -- mongo admin -u $USER -p $PASSWORD + +rs0:PRIMARY> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB + +rs0:PRIMARY> show users +{ + "_id" : "admin.root", + "userId" : UUID("12716b1d-9186-4710-a669-4a2050a548bc"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +rs0:PRIMARY> use newdb +switched to db newdb + +rs0:PRIMARY> db.movie.insert({"name":"batman"}); +WriteResult({ "nInserted" : 1 }) + +rs0:PRIMARY> db.movie.find().pretty() +{ "_id" : ObjectId("657182194f1f295758f449a5"), "name" : "batman" } + +rs0:PRIMARY> exit +bye + +``` + +Now, we are ready to backup this sample database. + +### Prepare Backend + +We are going to store our backed up data into a S3 bucket. At first, we need to create a secret with S3 credentials then we need to create a `BackupStorage` crd. If you want to use a different backend, please read the respective backend configuration doc from [here](https://stash.run/docs/latest/guides/backends/overview/). + +**Create Storage Secret:** + +Let's create a secret called `s3-secret` with access credentials to our desired S3 bucket, + +```console +$ echo -n '' > AWS_ACCESS_KEY_ID +$ echo -n '' > AWS_SECRET_ACCESS_KEY +$ kubectl create secret generic -n demo s3-secret \ + --from-file=./AWS_ACCESS_KEY_ID \ + --from-file=./AWS_SECRET_ACCESS_KEY +secret/s3-secret created +``` + +**Create BackupStorage:** + +Now, crete a `BackupStorage` using this secret. Below is the YAML of BackupStorage crd we are going to create, + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage-replicaset + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo-replicaset + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut +``` + +Let's create the `BackupStorage` we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupstorage-replicaset.yaml +backupstorage.storage.kubestash.com/s3-storage-replicaset created +``` + +Now, we are ready to backup our database to our desired backend. + +### Backup + +We have to create a `BackupConfiguration` targeting respective MongoDB crd of our desired database. Then KubeStash will create a CronJob to periodically backup the database. Before that we need to create an secret for encrypt data and retention policy. + +**Create Encryption Secret:** + +EncryptionSecret refers to the Secret containing the encryption key which will be used to encode/decode the backed up data. Let's create a secret called `encry-secret` + +```console +$ kubectl create secret generic encry-secret -n demo \ + --from-literal=RESTIC_PASSWORD='123' -n demo +secret/encry-secret created +``` + +**Create Retention Policy:** + +`RetentionPolicy` specifies how the old Snapshots should be cleaned up. This is a namespaced CRD.However, we can refer it from other namespaces as long as it is permitted via `.spec.usagePolicy`. Below is the YAML of the `RetentionPolicy` called `backup-rp` + +```console +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All +``` + +Let's create the RetentionPolicy we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/retentionpolicy.yaml +retentionpolicy.storage.kubestash.com/backup-rp created +``` + +**Create BackupConfiguration:** + +As we just create our encryption secret and retention policy, we are now ready to apply `BackupConfiguration` crd to take backup out database. + +Below is the YAML for `BackupConfiguration` crd to backup the `sample-mg-rs` database we have deployed earlier., + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mg-rs + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage-replicaset + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /replicaset + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup +``` + +Here, + +- `spec.target` specifies our targeted `MongoDB` database. +- `spec.backends` specifies `BackupStorage` information for storing data. +- `spec.sessions` specifies common session configurations for this backup +- `spec.sessions.schedule` specifies that we want to backup the database at 5 minutes interval. +- `spec.sessions.addon` refers to the `Addon` crd for backup task + +Let's create the `BackupConfiguration` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupconfiguration-replicaset.yaml +backupconfiguration.core.kubestash.com/mg created +``` + +**Verify Backup Setup Successful:** + +If everything goes well, the phase of the `BackupConfiguration` should be `Ready`. The `Ready` phase indicates that the backup setup is successful. Let's verify the `Phase` of the BackupConfiguration, + +```console +$ kubectl get backupconfiguration -n demo +NAME PHASE PAUSED AGE +mg Ready 85s +``` + +**Verify CronJob:** + +KubeStash will create a CronJob with the schedule specified in `spec.sessions.schedule` field of `BackupConfiguration` crd. + +Verify that the CronJob has been created using the following command, + +```console +$ kubectl get cronjob -n demo +NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE +trigger-mg-frequent */3 * * * * False 0 101s +``` + +**Wait for BackupSession:** + +The `trigger-mg-frequent` CronJob will trigger a backup on each schedule by creating a `BackpSession` crd. + +Wait for the next schedule. Run the following command to watch `BackupSession` crd, + +```console +$ kubectl get backupsession -n demo +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +mg-frequent-1701940862 BackupConfiguration mg Succeeded 3m16s +mg-frequent-1701941042 BackupConfiguration mg Running 16s +``` + +We can see above that the backup session has succeeded. Now, we are going to verify that the backed up data has been stored in the backend. + +**Verify Backup:** + +Once a backup is complete, KubeStash will update the respective `Snapshot` crd to reflect the backup. It will be created when a backup is triggered. Check that the `Snapshot` Phase to verify backup. + +```console +$ kubectl get snapshot -n demo +NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE VERIFICATION-STATUS AGE +s3-repo-mg-frequent-1701940862 s3-repo frequent 2023-12-07T09:21:07Z Delete Succeeded 3m53s +s3-repo-mg-frequent-1701941042 s3-repo frequent 2023-12-07T09:24:08Z Delete Succeeded 53s +``` + +KubeStash will also update the respective `Repository` crd to reflect the backup. Check that the repository `s3-repo` has been updated by the following command, + +```console +$ kubectl get repository -n demo s3-repo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +s3-repo true 2 2.883 KiB Ready 55s 8m5s +``` + +Now, if we navigate to the S3 bucket, we are going to see backed up data has been stored in `demo/replicaset/` directory as specified by `spec.sessions.repositories.directory` field of `BackupConfiguration` crd. + +> Note: KubeStash keeps all the backed up data encrypted. So, data in the backend will not make any sense until they are decrypted. + +## Restore MongoDB +You can restore your data into the same database you have backed up from or into a different database in the same cluster or a different cluster. In this section, we are going to show you how to restore in the same database which may be necessary when you have accidentally deleted any data from the running database. + +#### Stop Taking Backup of the Old Database: + +It's important to stop taking any further backup of the old database so that no backup is stored in our repository during restore process. KubeStash operator will automatically pause the `BackupConfiguration` when a `RestoreSession` is running. However if we want to pause the `BackupConfiguration` manually, we can do that by patching or using KubeStash CLI. + +Let's pause the `mg` BackupConfiguration by patching, +```bash +$ kubectl patch backupconfiguration -n demo mg --type="merge" --patch='{"spec": {"paused": true}}' +backupconfiguration.core.kubestash.com/mg patched +``` + +Now, wait for a moment. KubeStash will pause the BackupConfiguration. Verify that the BackupConfiguration has been paused, + +```console +$ kubectl get backupconfiguration -n demo mg +NAME PHASE PAUSED AGE +mg Ready true 11m +``` + +Notice the `PAUSED` column. Value `true` for this field means that the BackupConfiguration has been paused. + +#### Deploy New MongoDB Database For Restoring: + +We are going to deploy a new mongodb replicaset database for restoring backed up data. + +Below is the YAML of a sample `MongoDB` crd that we are going to create + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-rs-restore + namespace: demo +spec: + version: "4.2.3" + replicas: 3 + replicaSet: + name: rs0 + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset-restore.yaml +mongodb.kubedb.com/sample-mg-rs-restore created +``` + +Let's check if the database is ready to use, + +```console +$ kubectl get mg -n demo sample-mg-rs-restore +NAME VERSION STATUS AGE +sample-mg-rs-restore 4.2.3 Ready 2m45s +``` + +Let's verify all the databases of this `sample-mg-rs-restore` by exec into its pod + +```console +$ export USER=$(kubectl get secrets -n demo sample-mg-rs-restore-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo sample-mg-rs-restore-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo sample-mg-rs-restore-0 -- mongo admin -u $USER -p $PASSWORD + +rs0:PRIMARY> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB + +rs0:PRIMARY> show users +{ + "_id" : "admin.root", + "userId" : UUID("b8a10e3f-05a1-421c-87ca-7bdf16f99563"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +rs0:PRIMARY> exit +bye +``` + +As we can see no database named `newdb` exist in this new `sample-mg-rs-restore` database. + +#### Create RestoreSession: + +Now, we need to create a `RestoreSession` crd pointing to the `sample-mg-rs-restore` database. +Below is the YAML for the `RestoreSession` crd that we are going to create to restore the backed up data. + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-rs-restore + namespace: demo +spec: + target: + name: sample-mg-rs-restore + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore +``` + +Here, + +- `spec.dataSource.repository` specifies the `Repository` crd that holds the backend information where our backed up data has been stored. +- `spec.target` refers to the `MongoDB` crd for the `sample-mg-rs-restore` database. +- `spec.dataSource.snapshot` specifies that we are restoring from the latest backup snapshot of the `spec.dataSource.repository`. + +Let's create the `RestoreSession` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/restoresession-replicaset.yaml +restoresession.core.kubestash.com/mg-rs-restore created +``` + +Once, you have created the `RestoreSession` crd, KubeStash will create a job to restore. We can watch the `RestoreSession` phase to check if the restore process is succeeded or not. + +Run the following command to watch `RestoreSession` phase, + +```console +$ kubectl get restoresession -n demo mg-rs-restore -w +NAME REPOSITORY FAILURE-POLICY PHASE DURATION AGE +mg-rs-restore s3-repo Succeeded 9s 34s +``` + +So, we can see from the output of the above command that the restore process succeeded. + +#### Verify Restored Data: + +In this section, we are going to verify that the desired data has been restored successfully. We are going to connect to the `sample-mg-rs-restore` database and check whether the table we had created earlier is restored or not. + +Lets, exec into the database pod and list available tables, + +```console +$ kubectl exec -it -n demo sample-mg-rs-restore-0 -- mongo admin -u $USER -p $PASSWORD + +rs0:PRIMARY> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB +newdb 0.000GB + +rs0:PRIMARY> show users +{ + "_id" : "admin.root", + "userId" : UUID("b8a10e3f-05a1-421c-87ca-7bdf16f99563"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +rs0:PRIMARY> use newdb +switched to db newdb + +rs0:PRIMARY> db.movie.find().pretty() +{ "_id" : ObjectId("657182194f1f295758f449a5"), "name" : "batman" } + +rs0:PRIMARY> exit +bye +``` + +So, from the above output, we can see the database `newdb` that we had created earlier is restored into another new `MongoDB` database. + +## Cleanup + +To cleanup the Kubernetes resources created by this tutorial, run: + +```console +kubectl delete -n demo restoresession mg-rs-restore +kubectl delete -n demo backupconfiguration mg +kubectl delete -n demo mg sample-mg-rs +kubectl delete -n demo mg sample-mg-rs-restore +kubectl delete -n demo backupstorage s3-storage-replicaset +``` diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupconfiguration-replicaset.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupconfiguration-replicaset.yaml new file mode 100644 index 0000000000..7917991c3c --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupconfiguration-replicaset.yaml @@ -0,0 +1,34 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mg-rs + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage-replicaset + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /replicaset + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupstorage-replicaset.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupstorage-replicaset.yaml new file mode 100644 index 0000000000..81614714ba --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/backupstorage-replicaset.yaml @@ -0,0 +1,18 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage-replicaset + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo-replicaset + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset-restore.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset-restore.yaml new file mode 100644 index 0000000000..ae02e6bf9a --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset-restore.yaml @@ -0,0 +1,18 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-rs-restore + namespace: demo +spec: + version: "4.2.3" + replicas: 3 + replicaSet: + name: rs0 + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset.yaml new file mode 100644 index 0000000000..9a81d45a4d --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/mongodb-replicaset.yaml @@ -0,0 +1,18 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-rs + namespace: demo +spec: + version: "4.2.3" + replicas: 3 + replicaSet: + name: rs0 + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/restoresession-replicaset.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/restoresession-replicaset.yaml new file mode 100644 index 0000000000..56920b806c --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/restoresession-replicaset.yaml @@ -0,0 +1,21 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-rs-restore + namespace: demo +spec: + target: + name: sample-mg-rs-restore + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/retentionpolicy.yaml b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/retentionpolicy.yaml new file mode 100644 index 0000000000..3d327f9a71 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/replicaset/examples/retentionpolicy.yaml @@ -0,0 +1,12 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/_index.md b/docs/guides/mongodb/backup/kubestash/logical/sharding/_index.md new file mode 100644 index 0000000000..6eb3db4787 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/_index.md @@ -0,0 +1,607 @@ +--- +title: Backup & Restore Sharded MongoDB | KubeStash +description: Backup and restore sharded MongoDB database using KubeStash +menu: + docs_{{ .version }}: + identifier: guides-mongodb-backup-kubestash-logical-sharded + name: MongoDB Sharded Cluster + parent: guides-mongodb-backup-kubestash-logical + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Backup and Restore MongoDB database using KubeStash + +KubeStash v0.1.0+ supports backup and restoration of MongoDB databases. This guide will show you how you can backup and restore your MongoDB database with KubeStash. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. +- Install KubeDB in your cluster following the steps [here](/docs/setup/README.md). +- Install KubeStash Enterprise in your cluster following the steps [here(link needed)](). +- Install KubeStash `kubectl` plugin following the steps [here(link needed)](). +- If you are not familiar with how KubeStash backup and restore MongoDB databases, please check the following guide [here](/docs/guides/mongodb/backup/kubestash/overview/index.md). + +You have to be familiar with following custom resources: + +- [AppBinding](/docs/guides/mongodb/concepts/appbinding.md) +- [Function](https://stash.run/docs/latest/concepts/crds/function/) +- [Task](https://stash.run/docs/latest/concepts/crds/task/) +- [BackupConfiguration](https://stash.run/docs/latest/concepts/crds/backupconfiguration/) +- [RestoreSession](https://stash.run/docs/latest/concepts/crds/restoresession/) + +To keep things isolated, we are going to use a separate namespace called `demo` throughout this tutorial. Create `demo` namespace if you haven't created yet. + +```console +$ kubectl create ns demo +namespace/demo created +``` + +## Backup MongoDB + +This section will demonstrate how to backup MongoDB database. Here, we are going to deploy a MongoDB database using KubeDB. Then, we are going to backup this database into a S3 bucket. Finally, we are going to restore the backed up data into another MongoDB database. + +### Deploy Sample MongoDB Database + +Let's deploy a sample MongoDB database and insert some data into it. + +**Create MongoDB CRD:** + +Below is the YAML of a sample MongoDB crd that we are going to create for this tutorial: + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-sh + namespace: demo +spec: + version: 4.2.3 + shardTopology: + configServer: + replicas: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + mongos: + replicas: 2 + shard: + replicas: 3 + shards: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + terminationPolicy: WipeOut + +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding.yaml +mongodb.kubedb.com/sample-mg-sh created +``` + +KubeDB will deploy a MongoDB database according to the above specification. It will also create the necessary secrets and services to access the database. + +Let's check if the database is ready to use, + +```console +$ kubectl get mongodb -n demo sample-mg-sh +NAME VERSION STATUS AGE +sample-mg-sh 4.2.3 Ready 5m39s +``` + +The database is `Ready`. Verify that KubeDB has created a Secret and a Service for this database using the following commands, + +```console +$ kubectl get secret -n demo -l=app.kubernetes.io/instance=sample-mg-sh +NAME TYPE DATA AGE +sample-mg-sh-auth kubernetes.io/basic-auth 2 21m +sample-mg-sh-key Opaque 1 21m + +$ kubectl get service -n demo -l=app.kubernetes.io/instance=sample-mg-sh +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +sample-mg-sh ClusterIP 10.96.80.43 27017/TCP 21m +sample-mg-sh-configsvr-pods ClusterIP None 27017/TCP 21m +sample-mg-sh-mongos-pods ClusterIP None 27017/TCP 21m +sample-mg-sh-shard0-pods ClusterIP None 27017/TCP 21m +sample-mg-sh-shard1-pods ClusterIP None 27017/TCP 21m +sample-mg-sh-shard2-pods ClusterIP None 27017/TCP 21m +``` + +Here, we have to use service `sample-mg-sh` and secret `sample-mg-sh-auth` to connect with the database. + +**Insert Sample Data:** + +> Note: You can insert data into this `MongoDB` database using our [KubeDB CLI](https://kubedb.com/docs/latest/setup/install/kubectl_plugin/). + +For simplicity, we are going to exec into the database pod and create some sample data. At first, find out the database mongos pod using the following command, + +```console +$ kubectl get pods -n demo --selector="mongodb.kubedb.com/node.mongos=sample-mg-sh-mongos" +NAME READY STATUS RESTARTS AGE +sample-mg-sh-mongos-0 1/1 Running 0 21m +sample-mg-sh-mongos-1 1/1 Running 0 21m +``` + +Now, let's exec into the pod and create a table, + +```console +$ export USER=$(kubectl get secrets -n demo sample-mg-sh-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo sample-mg-sh-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo sample-mg-sh-mongos-0 -- mongo admin -u $USER -p $PASSWORD + +mongos> show dbs +admin 0.000GB +config 0.002GB +kubedb-system 0.000GB + +mongos> show users +{ + "_id" : "admin.root", + "userId" : UUID("61a02236-1e25-4f58-9b92-9a41a80726bc"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +mongos> use newdb +switched to db newdb + +mongos> db.movie.insert({"name":"batman"}); +WriteResult({ "nInserted" : 1 }) + +mongos> exit +bye + +``` + +Now, we are ready to backup this sample database. + +### Prepare Backend + +We are going to store our backed up data into a S3 bucket. At first, we need to create a secret with S3 credentials then we need to create a `BackupStorage` crd. If you want to use a different backend, please read the respective backend configuration doc from [here](https://stash.run/docs/latest/guides/backends/overview/). + +**Create Storage Secret:** + +Let's create a secret called `s3-secret` with access credentials to our desired S3 bucket, + +```console +$ echo -n '' > AWS_ACCESS_KEY_ID +$ echo -n '' > AWS_SECRET_ACCESS_KEY +$ kubectl create secret generic -n demo s3-secret \ + --from-file=./AWS_ACCESS_KEY_ID \ + --from-file=./AWS_SECRET_ACCESS_KEY +secret/s3-secret created +``` + +**Create BackupStorage:** + +Now, crete a `BackupStorage` using this secret. Below is the YAML of BackupStorage crd we are going to create, + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage-sharding + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo-sharding + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut +``` + +Let's create the `BackupStorage` we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupstorage-sharding.yaml +backupstorage.storage.kubestash.com/s3-storage-sharding created +``` + +Now, we are ready to backup our database to our desired backend. + +### Backup + +We have to create a `BackupConfiguration` targeting respective MongoDB crd of our desired database. Then KubeStash will create a CronJob to periodically backup the database. Before that we need to create an secret for encrypt data and retention policy. + +**Create Encryption Secret:** + +EncryptionSecret refers to the Secret containing the encryption key which will be used to encode/decode the backed up data. Let's create a secret called `encry-secret` + +```console +$ kubectl create secret generic encry-secret -n demo \ + --from-literal=RESTIC_PASSWORD='123' -n demo +secret/encry-secret created +``` + +**Create Retention Policy:** + +`RetentionPolicy` specifies how the old Snapshots should be cleaned up. This is a namespaced CRD.However, we can refer it from other namespaces as long as it is permitted via `.spec.usagePolicy`. Below is the YAML of the `RetentionPolicy` called `backup-rp` + +```console +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All +``` + +Let's create the RetentionPolicy we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/retentionpolicy.yaml +retentionpolicy.storage.kubestash.com/backup-rp created +``` + +**Create BackupConfiguration:** + +As we just create our encryption secret and retention policy, we are now ready to apply `BackupConfiguration` crd to take backup out database. + +Below is the YAML for `BackupConfiguration` crd to backup the `sample-mg-sh` database we have deployed earlier., + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mg-sh + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage-sharding + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /sharding + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup +``` + +Here, + +- `spec.target` specifies our targeted `MongoDB` database. +- `spec.backends` specifies `BackupStorage` information for storing data. +- `spec.sessions` specifies common session configurations for this backup +- `spec.sessions.schedule` specifies that we want to backup the database at 5 minutes interval. +- `spec.sessions.addon` refers to the `Addon` crd for backup task + +Let's create the `BackupConfiguration` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupconfiguration-sharding.yaml +backupconfiguration.core.kubestash.com/mg created +``` + +**Verify Backup Setup Successful:** + +If everything goes well, the phase of the `BackupConfiguration` should be `Ready`. The `Ready` phase indicates that the backup setup is successful. Let's verify the `Phase` of the BackupConfiguration, + +```console +$ kubectl get backupconfiguration -n demo +NAME PHASE PAUSED AGE +mg Ready 85s +``` + +**Verify CronJob:** + +KubeStash will create a CronJob with the schedule specified in `spec.sessions.schedule` field of `BackupConfiguration` crd. + +Verify that the CronJob has been created using the following command, + +```console +$ kubectl get cronjob -n demo +NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE +trigger-mg-frequent */3 * * * * False 0 101s +``` + +**Wait for BackupSession:** + +The `trigger-mg-frequent` CronJob will trigger a backup on each schedule by creating a `BackpSession` crd. + +Wait for the next schedule. Run the following command to watch `BackupSession` crd, + +```console +$ kubectl get backupsession -n demo +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +mg-frequent-1701950402 BackupConfiguration mg Succeeded 3m5s +mg-frequent-1701950582 BackupConfiguration mg Running 5s +``` + +We can see above that the backup session has succeeded. Now, we are going to verify that the backed up data has been stored in the backend. + +**Verify Backup:** + +Once a backup is complete, KubeStash will update the respective `Snapshot` crd to reflect the backup. It will be created when a backup is triggered. Check that the `Snapshot` Phase to verify backup. + +```console +$ kubectl get snapshot -n demo +NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE VERIFICATION-STATUS AGE +s3-repo-mg-frequent-1701950402 s3-repo frequent 2023-12-07T12:00:11Z Delete Succeeded 3m37s +s3-repo-mg-frequent-1701950582 s3-repo frequent 2023-12-07T12:03:08Z Delete Succeeded 37s +``` + +KubeStash will also update the respective `Repository` crd to reflect the backup. Check that the repository `s3-repo` has been updated by the following command, + +```console +$ kubectl get repository -n demo s3-repo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +s3-repo true 2 95.660 KiB Ready 41s 4m3s +``` + +Now, if we navigate to the S3 bucket, we are going to see backed up data has been stored in `demo/sharding/` directory as specified by `spec.sessions.repositories.directory` field of `BackupConfiguration` crd. + +> Note: KubeStash keeps all the backed up data encrypted. So, data in the backend will not make any sense until they are decrypted. + +## Restore MongoDB +You can restore your data into the same database you have backed up from or into a different database in the same cluster or a different cluster. In this section, we are going to show you how to restore in the same database which may be necessary when you have accidentally deleted any data from the running database. + +#### Stop Taking Backup of the Old Database: + +It's important to stop taking any further backup of the old database so that no backup is stored in our repository during restore process. KubeStash operator will automatically pause the `BackupConfiguration` when a `RestoreSession` is running. However if we want to pause the `BackupConfiguration` manually, we can do that by patching or using KubeStash CLI. + +Let's pause the `mg` BackupConfiguration by patching, +```bash +$ kubectl patch backupconfiguration -n demo mg --type="merge" --patch='{"spec": {"paused": true}}' +backupconfiguration.core.kubestash.com/mg patched +``` + +Now, wait for a moment. KubeStash will pause the BackupConfiguration. Verify that the BackupConfiguration has been paused, + +```console +$ kubectl get backupconfiguration -n demo mg +NAME PHASE PAUSED AGE +mg Ready true 11m +``` + +Notice the `PAUSED` column. Value `true` for this field means that the BackupConfiguration has been paused. + +#### Deploy New MongoDB Database For Restoring: + +We are going to deploy a new mongodb replicaset database for restoring backed up data. + +Below is the YAML of a sample `MongoDB` crd that we are going to create + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-sh-restore + namespace: demo +spec: + version: 4.2.3 + shardTopology: + configServer: + replicas: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + mongos: + replicas: 2 + shard: + replicas: 3 + shards: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + terminationPolicy: WipeOut +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding-restore.yaml +mongodb.kubedb.com/sample-mg-sh-restore created +``` + +Let's check if the database is ready to use, + +```console +$ kubectl get mg -n demo sample-mg-sh-restore +NAME VERSION STATUS AGE +sample-mg-sh-restore 4.2.3 Ready 7m47s +``` + +Let's verify all the databases of this `sample-mg-sh-restore` by exec into its mongos pod + +```console +$ export USER=$(kubectl get secrets -n demo sample-mg-sh-restore-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo sample-mg-sh-restore-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo sample-mg-sh-restore-mongos-0 -- mongo admin -u $USER -p $PASSWORD + +mongos> show dbs +admin 0.000GB +config 0.002GB +kubedb-system 0.000GB + +mongos> show users +{ + "_id" : "admin.root", + "userId" : UUID("4400d0cc-bba7-4626-bf5b-7521fca30ff9"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +mongos> exit +bye +``` + +As we can see no database named `newdb` exist in this new `sample-mg-sh-restore` database. + +#### Create RestoreSession: + +Now, we need to create a `RestoreSession` crd pointing to the `sample-mg-sh-restore` database. +Below is the YAML for the `RestoreSession` crd that we are going to create to restore the backed up data. + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-sh-restore + namespace: demo +spec: + target: + name: sample-mg-sh-restore + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore +``` + +Here, + +- `spec.dataSource.repository` specifies the `Repository` crd that holds the backend information where our backed up data has been stored. +- `spec.target` refers to the `MongoDB` crd for the `sample-mg-sh-restore` database. +- `spec.dataSource.snapshot` specifies that we are restoring from the latest backup snapshot of the `spec.dataSource.repository`. + +Let's create the `RestoreSession` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/restoresession-sharding.yaml +restoresession.core.kubestash.com/mg-sh-restore created +``` + +Once, you have created the `RestoreSession` crd, KubeStash will create a job to restore. We can watch the `RestoreSession` phase to check if the restore process is succeeded or not. + +Run the following command to watch `RestoreSession` phase, + +```console +$ kubectl get restoresession -n demo mg-sh-restore -w +NAME REPOSITORY FAILURE-POLICY PHASE DURATION AGE +mg-sh-restore s3-repo Succeeded 15s 48s +``` + +So, we can see from the output of the above command that the restore process succeeded. + +#### Verify Restored Data: + +In this section, we are going to verify that the desired data has been restored successfully. We are going to connect to the `sample-mg-sh-restore` database and check whether the table we had created earlier is restored or not. + +Lets, exec into the database's mongos pod and list available tables, + +```console +$ kubectl exec -it -n demo sample-mg-sh-restore-mongos-0 -- mongo admin -u $USER -p $PASSWORD + +mongos> show dbs +admin 0.000GB +config 0.002GB +kubedb-system 0.000GB +newdb 0.000GB + +mongos> show users +{ + "_id" : "admin.root", + "userId" : UUID("4400d0cc-bba7-4626-bf5b-7521fca30ff9"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +mongos> use newdb +switched to db newdb + +mongos> db.movie.find().pretty() +{ "_id" : ObjectId("6571b0465bde128e0c4d7caa"), "name" : "batman" } + +mongos> exit +bye +``` + +So, from the above output, we can see the database `newdb` that we had created earlier is restored into another new `MongoDB` database. + +## Cleanup + +To cleanup the Kubernetes resources created by this tutorial, run: + +```console +kubectl delete -n demo restoresession mg-sh-restore +kubectl delete -n demo backupconfiguration mg +kubectl delete -n demo mg sample-mg-sh +kubectl delete -n demo mg sample-mg-sh-restore +kubectl delete -n demo backupstorage s3-storage-sharding +``` diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupconfiguration-sharding.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupconfiguration-sharding.yaml new file mode 100644 index 0000000000..6cc56b8028 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupconfiguration-sharding.yaml @@ -0,0 +1,34 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mg-sh + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage-sharding + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /sharding + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupstorage-sharding.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupstorage-sharding.yaml new file mode 100644 index 0000000000..39c06aeca2 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/backupstorage-sharding.yaml @@ -0,0 +1,18 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage-sharding + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo-sharding + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding-restore.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding-restore.yaml new file mode 100644 index 0000000000..18fd34b3e5 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding-restore.yaml @@ -0,0 +1,26 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-sh-restore + namespace: demo +spec: + version: 4.2.3 + shardTopology: + configServer: + replicas: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + mongos: + replicas: 2 + shard: + replicas: 3 + shards: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + terminationPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding.yaml new file mode 100644 index 0000000000..09dfc2ad12 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/mongodb-sharding.yaml @@ -0,0 +1,26 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mg-sh + namespace: demo +spec: + version: 4.2.3 + shardTopology: + configServer: + replicas: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + mongos: + replicas: 2 + shard: + replicas: 3 + shards: 3 + storage: + resources: + requests: + storage: 1Gi + storageClassName: standard + terminationPolicy: WipeOut diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/restoresession-sharding.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/restoresession-sharding.yaml new file mode 100644 index 0000000000..d0d8614d7d --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/restoresession-sharding.yaml @@ -0,0 +1,21 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-sh-restore + namespace: demo +spec: + target: + name: sample-mg-sh-restore + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/retentionpolicy.yaml b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/retentionpolicy.yaml new file mode 100644 index 0000000000..3d327f9a71 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/sharding/examples/retentionpolicy.yaml @@ -0,0 +1,12 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/_index.md b/docs/guides/mongodb/backup/kubestash/logical/standalone/_index.md new file mode 100644 index 0000000000..3d84912808 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/_index.md @@ -0,0 +1,586 @@ +--- +title: Backup & Restore Standalone MongoDB | KubeStash +description: Backup and restore standalone MongoDB database using KubeStash +menu: + docs_{{ .version }}: + identifier: guides-mongodb-backup-kubestash-logical-standalone + name: Standalone MongoDB + parent: guides-mongodb-backup-kubestash-logical + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Backup and Restore MongoDB database using KubeStash + +KubeStash v0.1.0+ supports backup and restoration of MongoDB databases. This guide will show you how you can backup and restore your MongoDB database with KubeStash. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. +- Install KubeDB in your cluster following the steps [here](/docs/setup/README.md). +- Install KubeStash Enterprise in your cluster following the steps [here(link needed)](). +- Install KubeStash `kubectl` plugin following the steps [here(link needed)](). +- If you are not familiar with how KubeStash backup and restore MongoDB databases, please check the following guide [here](/docs/guides/mongodb/backup/kubestash/overview/index.md). + +You have to be familiar with following custom resources: + +- [AppBinding](/docs/guides/mongodb/concepts/appbinding.md) +- [Function](https://stash.run/docs/latest/concepts/crds/function/) +- [Task](https://stash.run/docs/latest/concepts/crds/task/) +- [BackupConfiguration](https://stash.run/docs/latest/concepts/crds/backupconfiguration/) +- [RestoreSession](https://stash.run/docs/latest/concepts/crds/restoresession/) + +To keep things isolated, we are going to use a separate namespace called `demo` throughout this tutorial. Create `demo` namespace if you haven't created yet. + +```console +$ kubectl create ns demo +namespace/demo created +``` + +## Backup MongoDB + +This section will demonstrate how to backup MongoDB database. Here, we are going to deploy a MongoDB database using KubeDB. Then, we are going to backup this database into a S3 bucket. Finally, we are going to restore the backed up data into another MongoDB database. + +### Deploy Sample MongoDB Database + +Let's deploy a sample MongoDB database and insert some data into it. + +**Create MongoDB CRD:** + +Below is the YAML of a sample MongoDB crd that we are going to create for this tutorial: + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mongodb + namespace: demo +spec: + version: "4.2.3" + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb.yaml +mongodb.kubedb.com/sample-mongodb created +``` + +KubeDB will deploy a MongoDB database according to the above specification. It will also create the necessary secrets and services to access the database. + +Let's check if the database is ready to use, + +```console +$ kubectl get mg -n demo sample-mongodb +NAME VERSION STATUS AGE +sample-mongodb 4.2.3 Ready 2m9s +``` + +The database is `Ready`. Verify that KubeDB has created a Secret and a Service for this database using the following commands, + +```console +$ kubectl get secret -n demo -l=app.kubernetes.io/instance=sample-mongodb +NAME TYPE DATA AGE +sample-mongodb-auth Opaque 2 2m28s + +$ kubectl get service -n demo -l=app.kubernetes.io/instance=sample-mongodb +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +sample-mongodb ClusterIP 10.107.58.222 27017/TCP 2m48s +sample-mongodb-gvr ClusterIP None 27017/TCP 2m48s +``` + +Here, we have to use service `sample-mongodb` and secret `sample-mongodb-auth` to connect with the database. + +**Insert Sample Data:** + +> Note: You can insert data into this `MongoDB` database using our [KubeDB CLI](https://kubedb.com/docs/latest/setup/install/kubectl_plugin/). + +For simplicity, we are going to exec into the database pod and create some sample data. At first, find out the database pod using the following command, + +```console +$ kubectl get pods -n demo --selector="app.kubernetes.io/instance=sample-mongodb" +NAME READY STATUS RESTARTS AGE +sample-mongodb-0 1/1 Running 0 12m +``` + +Now, let's exec into the pod and create a table, + +```console +$ export USER=$(kubectl get secrets -n demo sample-mongodb-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo sample-mongodb-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo sample-mongodb-0 -- mongo admin -u $USER -p $PASSWORD + +> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB + +> show users +{ + "_id" : "admin.root", + "userId" : UUID("b82f8a34-1fc4-4ffe-b616-c6ffa278ecc8"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +> use newdb +switched to db newdb + +> db.movie.insert({"name":"batman"}); +WriteResult({ "nInserted" : 1 }) + +> db.movie.find().pretty() +{ "_id" : ObjectId("5d19d1cdc93d828f44e37735"), "name" : "batman" } + +> exit +bye +``` + +Now, we are ready to backup this sample database. + +### Prepare Backend + +We are going to store our backed up data into a S3 bucket. At first, we need to create a secret with S3 credentials then we need to create a `BackupStorage` crd. If you want to use a different backend, please read the respective backend configuration doc from [here](https://stash.run/docs/latest/guides/backends/overview/). + +**Create Storage Secret:** + +Let's create a secret called `s3-secret` with access credentials to our desired S3 bucket, + +```console +$ echo -n '' > AWS_ACCESS_KEY_ID +$ echo -n '' > AWS_SECRET_ACCESS_KEY +$ kubectl create secret generic -n demo s3-secret \ + --from-file=./AWS_ACCESS_KEY_ID \ + --from-file=./AWS_SECRET_ACCESS_KEY +secret/s3-secret created +``` + +**Create BackupStorage:** + +Now, crete a `BackupStorage` using this secret. Below is the YAML of BackupStorage crd we are going to create, + +```yaml +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut +``` + +Let's create the `BackupStorage` we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupstorage.yaml +storage.kubestash.com/s3-storage created +``` + +Now, we are ready to backup our database to our desired backend. + +### Backup + +We have to create a `BackupConfiguration` targeting respective MongoDB crd of our desired database. Then KubeStash will create a CronJob to periodically backup the database. Before that we need to create an secret for encrypt data and retention policy. + +**Create Encryption Secret:** + +EncryptionSecret refers to the Secret containing the encryption key which will be used to encode/decode the backed up data. Let's create a secret called `encry-secret` + +```console +$ kubectl create secret generic encry-secret -n demo \ + --from-literal=RESTIC_PASSWORD='123' -n demo +secret/encry-secret created +``` + +**Create Retention Policy:** + +`RetentionPolicy` specifies how the old Snapshots should be cleaned up. This is a namespaced CRD.However, we can refer it from other namespaces as long as it is permitted via `.spec.usagePolicy`. Below is the YAML of the `RetentionPolicy` called `backup-rp` + +```console +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All +``` + +Let's create the RetentionPolicy we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/retentionpolicy.yaml +retentionpolicy.storage.kubestash.com/backup-rp created +``` + +**Create BackupConfiguration:** + +As we just create our encryption secret and retention policy, we are now ready to apply `BackupConfiguration` crd to take backup out database. + +Below is the YAML for `BackupConfiguration` crd to backup the `sample-mongodb` database we have deployed earlier., + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mongodb + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /mongodb + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup +``` + +Here, + +- `spec.target` specifies our targeted `MongoDB` database. +- `spec.backends` specifies `BackupStorage` information for storing data. +- `spec.sessions` specifies common session configurations for this backup +- `spec.sessions.schedule` specifies that we want to backup the database at 5 minutes interval. +- `spec.sessions.addon` refers to the `Addon` crd for backup task + +Let's create the `BackupConfiguration` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupconfiguration.yaml +backupconfiguration.core.kubestash.com/mg created +``` + +**Verify Backup Setup Successful:** + +If everything goes well, the phase of the `BackupConfiguration` should be `Ready`. The `Ready` phase indicates that the backup setup is successful. Let's verify the `Phase` of the BackupConfiguration, + +```console +$ kubectl get backupconfiguration -n demo +NAME PHASE PAUSED AGE +mg Ready 85s +``` + +**Verify CronJob:** + +KubeStash will create a CronJob with the schedule specified in `spec.sessions.schedule` field of `BackupConfiguration` crd. + +Verify that the CronJob has been created using the following command, + +```console +$ kubectl get cronjob -n demo +NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE +trigger-mg-frequent */3 * * * * False 0 101s +``` + +**Wait for BackupSession:** + +The `trigger-mg-frequent` CronJob will trigger a backup on each schedule by creating a `BackpSession` crd. + +Wait for the next schedule. Run the following command to watch `BackupSession` crd, + +```console +$ kubectl get backupsession -n demo +NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE +mg-frequent-1701923402 BackupConfiguration mg Succeeded 3m4s +mg-frequent-1701923582 BackupConfiguration mg Running 4s +``` + +We can see above that the backup session has succeeded. Now, we are going to verify that the backed up data has been stored in the backend. + +**Verify Backup:** + +Once a backup is complete, KubeStash will update the respective `Snapshot` crd to reflect the backup. It will be created when a backup is triggered. Check that the `Snapshot` Phase to verify backup. + +```console +$ kubectl get snapshot -n demo +NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE VERIFICATION-STATUS AGE +s3-repo-mg-frequent-1701923402 s3-repo frequent 2023-12-07T04:30:10Z Delete Succeeded 3m25s +s3-repo-mg-frequent-1701923582 s3-repo frequent 2023-12-07T04:33:06Z Delete Succeeded 25s +``` + + +KubeStash will also update the respective `Repository` crd to reflect the backup. Check that the repository `s3-repo` has been updated by the following command, + +```console +$ kubectl get repository -n demo s3-repo +NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE +s3-repo true 2 2.613 KiB Ready 2m42s 8m38s +``` + +Now, if we navigate to the S3 bucket, we are going to see backed up data has been stored in `demo/mongodb/` directory as specified by `spec.sessions.repositories.directory` field of `BackupConfiguration` crd. + +> Note: KubeStash keeps all the backed up data encrypted. So, data in the backend will not make any sense until they are decrypted. + +## Restore MongoDB +You can restore your data into the same database you have backed up from or into a different database in the same cluster or a different cluster. In this section, we are going to show you how to restore in the same database which may be necessary when you have accidentally deleted any data from the running database. + +#### Stop Taking Backup of the Old Database: + +It's important to stop taking any further backup of the old database so that no backup is stored in our repository during restore process. KubeStash operator will automatically pause the `BackupConfiguration` when a `RestoreSession` is running. However if we want to pause the `BackupConfiguration` manually, we can do that by patching or using KubeStash CLI. + +Let's pause the `mg` BackupConfiguration by patching, +```bash +$ kubectl patch backupconfiguration -n demo mg --type="merge" --patch='{"spec": {"paused": true}}' +backupconfiguration.core.kubestash.com/mg patched +``` + +Now, wait for a moment. KubeStash will pause the BackupConfiguration. Verify that the BackupConfiguration has been paused, + +```console +$ kubectl get backupconfiguration -n demo mg +NAME PHASE PAUSED AGE +mg Ready true 26m +``` + +Notice the `PAUSED` column. Value `true` for this field means that the BackupConfiguration has been paused. + +#### Deploy New MongoDB Database For Restoring: + +We are going to deploy a new mongodb standalone database for restoring backed up data. + +Below is the YAML of a sample `MongoDB` crd that we are going to create + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: restore-mongodb + namespace: demo +spec: + version: "4.2.3" + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut +``` + +Create the above `MongoDB` crd, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb-restore.yaml +mongodb.kubedb.com/restore-mongodb created +``` + +Let's check if the database is ready to use, + +```console +$ kubectl get mg -n demo restore-mongodb +NAME VERSION STATUS AGE +restore-mongodb 4.2.3 Ready 3m30s +``` + +Let's verify all the databases of this `restore-mongodb` by exec into its pod + +```console +$ export USER=$(kubectl get secrets -n demo restore-mongodb-auth -o jsonpath='{.data.\username}' | base64 -d) + +$ export PASSWORD=$(kubectl get secrets -n demo restore-mongodb-auth -o jsonpath='{.data.\password}' | base64 -d) + +$ kubectl exec -it -n demo restore-mongodb-0 -- mongo admin -u $USER -p $PASSWORD + +> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB + +> show users +{ + "_id" : "admin.root", + "userId" : UUID("a4decc6b-959a-434d-9e4b-19cb9bfa783b"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +> exit +bye +``` + +As we can see no database named `newdb` exist in this new `restore-mongodb` database. + +#### Create RestoreSession: + +Now, we need to create a `RestoreSession` crd pointing to the `restore-mongodb` database. +Below is the YAML for the `RestoreSession` crd that we are going to create to restore the backed up data. + +```yaml +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-restore + namespace: demo +spec: + target: + name: restore-mongodb + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore +``` + +Here, + +- `spec.dataSource.repository` specifies the `Repository` crd that holds the backend information where our backed up data has been stored. +- `spec.target` refers to the `MongoDB` crd for the `restore-mongodb` database. +- `spec.dataSource.snapshot` specifies that we are restoring from the latest backup snapshot of the `spec.dataSource.repository`. + +Let's create the `RestoreSession` crd we have shown above, + +```console +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/restoresession.yaml +restoresession.core.kubestash.com/mg-restore created +``` + +Once, you have created the `RestoreSession` crd, KubeStash will create a job to restore. We can watch the `RestoreSession` phase to check if the restore process is succeeded or not. + +Run the following command to watch `RestoreSession` phase, + +```console +$ kubectl get restoresession -n demo sample-mongodb-restore -w +NAME REPOSITORY FAILURE-POLICY PHASE DURATION AGE +mg-restore s3-repo Succeeded 8s 49s +``` + +So, we can see from the output of the above command that the restore process succeeded. + +#### Verify Restored Data: + +In this section, we are going to verify that the desired data has been restored successfully. We are going to connect to the `restore-mongodb` database and check whether the table we had created earlier is restored or not. + +Lets, exec into the database pod and list available tables, + +```console +$ kubectl exec -it -n demo restore-mongodb-0 -- mongo admin -u $USER -p $PASSWORD + +> show dbs +admin 0.000GB +config 0.000GB +kubedb-system 0.000GB +local 0.000GB +newdb 0.000GB + +> show users +{ + "_id" : "admin.root", + "userId" : UUID("a4decc6b-959a-434d-9e4b-19cb9bfa783b"), + "user" : "root", + "db" : "admin", + "roles" : [ + { + "role" : "root", + "db" : "admin" + } + ], + "mechanisms" : [ + "SCRAM-SHA-1", + "SCRAM-SHA-256" + ] +} + +> use newdb +switched to db newdb + +> db.movie.find().pretty() +{ "_id" : ObjectId("5d19d1cdc93d828f44e37735"), "name" : "batman" } + +> exit +bye +``` + +So, from the above output, we can see the database `newdb` that we had created earlier is restored into another new `MongoDB` database. + +## Cleanup + +To cleanup the Kubernetes resources created by this tutorial, run: + +```console +kubectl delete -n demo restoresession mg-restore +kubectl delete -n demo backupconfiguration mg +kubectl delete -n demo mg sample-mongodb +kubectl delete -n demo mg restore-mongodb +kubectl delete -n demo backupstorage s3-storage +``` diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupconfiguration.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupconfiguration.yaml new file mode 100644 index 0000000000..5d15c0a99f --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupconfiguration.yaml @@ -0,0 +1,34 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: BackupConfiguration +metadata: + name: mg + namespace: demo +spec: + target: + apiGroup: kubedb.com + kind: MongoDB + namespace: demo + name: sample-mongodb + backends: + - name: s3-backend + storageRef: + namespace: demo + name: s3-storage + retentionPolicy: + name: backup-rp + namespace: demo + sessions: + - name: frequent + scheduler: + schedule: "*/3 * * * *" + repositories: + - name: s3-repo + backend: s3-backend + directory: /mongodb + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackup \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupstorage.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupstorage.yaml new file mode 100644 index 0000000000..0eeb7f1ab7 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/backupstorage.yaml @@ -0,0 +1,18 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: BackupStorage +metadata: + name: s3-storage + namespace: demo +spec: + storage: + provider: s3 + s3: + endpoint: us-east-1.linodeobjects.com + bucket: kubestash-testing + region: us-east-1 + prefix: demo + secret: s3-secret + usagePolicy: + allowedNamespaces: + from: All + deletionPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb-restore.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb-restore.yaml new file mode 100644 index 0000000000..005a946361 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb-restore.yaml @@ -0,0 +1,16 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: restore-mongodb + namespace: demo +spec: + version: "4.2.3" + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb.yaml new file mode 100644 index 0000000000..3b14c34e75 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/mongodb.yaml @@ -0,0 +1,16 @@ +apiVersion: kubedb.com/v1alpha2 +kind: MongoDB +metadata: + name: sample-mongodb + namespace: demo +spec: + version: "4.2.3" + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + terminationPolicy: WipeOut \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/restoresession.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/restoresession.yaml new file mode 100644 index 0000000000..1020635be4 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/restoresession.yaml @@ -0,0 +1,21 @@ +apiVersion: core.kubestash.com/v1alpha1 +kind: RestoreSession +metadata: + name: mg-restore + namespace: demo +spec: + target: + name: restore-mongodb + namespace: demo + apiGroup: kubedb.com + kind: MongoDB + dataSource: + snapshot: latest + repository: s3-repo + encryptionSecret: + name: encry-secret + namespace: demo + addon: + name: mongodb-addon + tasks: + - name: LogicalBackupRestore \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/retentionpolicy.yaml b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/retentionpolicy.yaml new file mode 100644 index 0000000000..3d327f9a71 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/logical/standalone/examples/retentionpolicy.yaml @@ -0,0 +1,12 @@ +apiVersion: storage.kubestash.com/v1alpha1 +kind: RetentionPolicy +metadata: + name: backup-rp + namespace: demo +spec: + maxRetentionPeriod: 2mo + successfulSnapshots: + last: 10 + usagePolicy: + allowedNamespaces: + from: All \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/_index.md b/docs/guides/mongodb/backup/kubestash/overview/_index.md index 07c97376d8..599c682ba3 100644 --- a/docs/guides/mongodb/backup/kubestash/overview/_index.md +++ b/docs/guides/mongodb/backup/kubestash/overview/_index.md @@ -7,7 +7,6 @@ menu: parent: guides-mongodb-backup-kubestash weight: 10 menu_name: docs_{{ .version }} -section_menu_id: guides --- > New to KubeDB? Please start [here](/docs/README.md). @@ -25,4 +24,141 @@ The following diagram shows how KubeStash takes backup of a MongoDB database. Op
MongoDB Backup Overview
Fig: MongoDB Backup Overview
-
\ No newline at end of file + + +The backup process consists of the following steps: + +1. At first, a user creates a secret with access credentials of the backend where the backed up data will be stored. + +2. Then, she creates a `BackupStorage` crd that specifies the backend information along with the secret that holds the credentials to access the backend. + +3. Then, she creates a `BackupConfiguration` crd targeting the crd of the desired `MongoDB` database. The `BackupConfiguration` object also specifies the `Sessions` to use to backup the database. + +4. KubeStash operator watches for `BackupConfiguration` crd. + +5. Once KubeStash operator finds a `BackupConfiguration` crd, it creates a CronJob with the schedule specified in `BackupConfiguration` object to trigger backup periodically. + +6. On the next scheduled slot, the CronJob triggers a backup by creating a `BackupSession` crd. + +7. KubeStash operator also watches for `BackupSession` crd. + +8. When it finds a `BackupSession` object, It creates a `Snapshot` for holding backup information. + +9. KubeStash resolves the respective `Addon` and `Function` and prepares a Job definition to backup. + +10. Then, it creates the Job to backup the targeted database. + +11. The backup Job reads necessary information to connect with the database from the `AppBinding` crd of the targated `MongoDB` database. It also reads backend information and access credentials from `BackupStorage` crd and Storage Secret respectively through `Backend` section of `BackupConfiguration` crd + +12. Then, the Job dumps the targeted database and uploads the output to the backend. KubeStash pipes the output of dump command to uploading process. Hence, backup Job does not require a large volume to hold the entire dump output. + +13. Finally, when the backup is complete, the Job sends Prometheus metrics to the Pushgateway running inside KubeStash operator pod. It also updates the `BackupSession` and `Snapshot` status to reflect the backup procedure. + +### Backup Different MongoDB Configurations + +This section will show you how backup works for different MongoDB configurations. + +#### Standalone MongoDB + +For a standalone MongoDB database, the backup job directly dumps the database using `mongodump` and pipe the output to the backup process. + +
+ Standalone MongoDB Backup Overview +
Fig: Standalone MongoDB Backup
+
+ +#### MongoDB ReplicaSet Cluster + +For MongoDB ReplicaSet cluster, KubeStash takes backup from one of the secondary replicas. The backup process consists of the following steps: + +1. Identify a secondary replica. +2. Lock the secondary replica. +3. Backup the secondary replica. +4. Unlock the secondary replica. + +
+ MongoDB ReplicaSet Cluster Backup Overview +
Fig: MongoDB ReplicaSet Cluster Backup
+
+ +#### MongoDB Sharded Cluster + +For MongoDB sharded cluster, KubeStash takes backup of the individual shards as well as the config server. KubeStash takes backup from a secondary replica of the shards and the config server. If there is no secondary replica then KubeStash will take backup from the primary replica. The backup process consists of the following steps: + +1. Disable balancer. +2. Lock config server. +3. Identify a secondary replica for each shard. +4. Lock the secondary replica. +5. Run backup on the secondary replica. +6. Unlock the secondary replica. +7. Unlock config server. +8. Enable balancer. + +
+ MongoDB Sharded Cluster Backup Overview +
Fig: MongoDB Sharded Cluster Backup
+
+ +## How Restore Process Works + +The following diagram shows how KubeStash restores backed up data into a MongoDB database. Open the image in a new tab to see the enlarged version. + +
+ Database Restore Overview +
Fig: MongoDB Restore Process Overview
+
+ +The restore process consists of the following steps: + +1. At first, a user creates a `RestoreSession` crd targeting the `AppBinding` of the desired database where the backed up data will be restored. It also specifies the `Repository` crd which holds the backend information and the `Task` to use to restore the target. + +2. KubeStash operator watches for `RestoreSession` object. + +3. Once it finds a `RestoreSession` object, it resolves the respective `Task` and `Function` and prepares a Job definition to restore. + +4. Then, it creates the Job to restore the target. + +5. The Job reads necessary information to connect with the database from respective `AppBinding` crd. It also reads backend information and access credentials from `Repository` crd and Storage Secret respectively. + +6. Then, the job downloads the backed up data from the backend and injects into the desired database. KubeStash pipes the downloaded data to the respective database tool to inject into the database. Hence, restore job does not require a large volume to download entire backup data inside it. + +7. Finally, when the restore process is complete, the Job sends Prometheus metrics to the Pushgateway and update the `RestoreSession` status to reflect restore completion. + +### Restoring Different MongoDB Configurations + +This section will show you restore process works for different MongoDB configurations. + +#### Standalone MongoDB + +For a standalone MongoDB database, the restore job downloads the backed up data from the backend and pipe the downloaded data to `mongorestore` command which inserts the data into the desired MongoDB database. + +
+ Standalone MongoDB Restore Overview +
Fig: Standalone MongoDB Restore
+
+ +#### MongoDB ReplicaSet Cluster + +For MongoDB ReplicaSet cluster, KubeStash identifies the primary replica and restore into it. + +
+ MongoDB ReplicaSet Cluster Restore Overview +
Fig: MongoDB ReplicaSet Cluster Restore
+
+ +#### MongoDB Sharded Cluster + +For MongoDB sharded cluster, KubeStash identifies the primary replica of each shard as well as the config server and restore respective backed up data into them. + +
+ MongoDB Sharded Cluster Restore +
Fig: MongoDB Sharded Cluster Restore
+
+ +## Next Steps + +- Backup a standalone MongoDB databases using Stash following the guide from [here](/docs/guides/mongodb/backup/kubestash/logical/standalone/index.md). +- Backup a MongoDB Replicaset cluster using Stash following the guide from [here](/docs/guides/mongodb/backup/kubestash/logical/replicaset/index.md). +- Backup a sharded MongoDB cluster using Stash following the guide from [here](/docs/guides/mongodb/backup/kubestash/logical/sharding/index.md). + + diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_backup.svg b/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_backup.svg new file mode 100644 index 0000000000..e41cdcc771 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_backup.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
R1
R1
R2
R2
R3
R3
MongoDB
ReplicaSet
Cluster
MongoDB...
Primary
Primary
Secondary
Secondary
Secondary
Secondary
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_restore.svg b/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_restore.svg new file mode 100644 index 0000000000..c6cfa868ba --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/replicaset_restore.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
R1
R1
R2
R2
R3
R3
MongoDB
ReplicaSet
Cluster
MongoDB...
Primary
Primary
Secondary
Secondary
Secondary
Secondary
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/restore_overview.svg b/docs/guides/mongodb/backup/kubestash/overview/images/restore_overview.svg new file mode 100644 index 0000000000..2b587b058a --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/restore_overview.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/sharded_backup.svg b/docs/guides/mongodb/backup/kubestash/overview/images/sharded_backup.svg new file mode 100644 index 0000000000..3e0b4cb461 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/sharded_backup.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
R1
R1
R2
R2
R3
R3
MongoDB
ReplicaSet
Cluster
MongoDB...
Primary
Primary
Secondary
Secondary
Secondary
Secondary
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/sharded_restore.svg b/docs/guides/mongodb/backup/kubestash/overview/images/sharded_restore.svg new file mode 100644 index 0000000000..25b7126c02 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/sharded_restore.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
R1
R1
R2
R2
R3
R3
MongoDB
ReplicaSet
Cluster
MongoDB...
Primary
Primary
Secondary
Secondary
Secondary
Secondary
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/standalone_backup.svg b/docs/guides/mongodb/backup/kubestash/overview/images/standalone_backup.svg new file mode 100644 index 0000000000..47eb24665c --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/standalone_backup.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
MongoDB
MongoDB
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file diff --git a/docs/guides/mongodb/backup/kubestash/overview/images/standalone_restore.svg b/docs/guides/mongodb/backup/kubestash/overview/images/standalone_restore.svg new file mode 100644 index 0000000000..1c72f07bb7 --- /dev/null +++ b/docs/guides/mongodb/backup/kubestash/overview/images/standalone_restore.svg @@ -0,0 +1,4 @@ + + + +
KubeStash
Backup Job
KubeStash...
R1
R1
R2
R2
R3
R3
MongoDB
ReplicaSet
Cluster
MongoDB...
Primary
Primary
Secondary
Secondary
Secondary
Secondary
mongodump
mongodumpText is not SVG - cannot display \ No newline at end of file