From bae550271ed7a5ba5e7f85a1d1047ce879d52591 Mon Sep 17 00:00:00 2001 From: Theresa Kamerman Date: Wed, 11 Dec 2024 10:01:15 -0800 Subject: [PATCH] Update DB Migration docs - Move old docs to unlisted subpage --- .../advanced-database-migrations-OLD.mdx | 192 ++++++++++++++++++ .../advanced-database-migrations.mdx | 137 ++++--------- 2 files changed, 227 insertions(+), 102 deletions(-) create mode 100644 docs/deployment/advanced-database-migrations-OLD.mdx diff --git a/docs/deployment/advanced-database-migrations-OLD.mdx b/docs/deployment/advanced-database-migrations-OLD.mdx new file mode 100644 index 0000000..60f4ebe --- /dev/null +++ b/docs/deployment/advanced-database-migrations-OLD.mdx @@ -0,0 +1,192 @@ +# Advanced - Database Migrations pre-Aerie v3.3.0 + +:::info Note +For Aerie deployments AFTER to v3.3.0, please reference [this page](../advanced-database-migrations) +for accurate information as to how the migration script works. +::: + +To aid in migrating between database versions we provide a Python script called [aerie_db_migration.py](https://github.com/NASA-AMMOS/aerie/blob/develop/deployment/aerie_db_migration). In order to run it, the following software is required: + +- [Python 3.9](https://www.python.org/downloads/) or later +- [Hasura CLI 2.35.1](https://hasura.io/docs/latest/hasura-cli/install-hasura-cli/) or later + +Additional Python package requirements can be installed by running the following command in the [deployment](https://github.com/NASA-AMMOS/aerie/tree/develop/deployment) directory: + +```sh +python -m pip install -r requirements.txt +``` + +Once the prerequisite software has been installed, the script can be executed by running the following command from inside the Aerie [deployment](https://github.com/NASA-AMMOS/aerie/tree/develop/deployment) directory: + +```sh +python aerie_db_migration.py +``` + +Alternatively, you can add the script to your `PATH` and run it from anywhere with the command `aerie_db_migration.py`: + +```sh +export PATH="$HOME/path/to/deployment:$PATH" +``` + +The script expects all migration files present on the server to also exist on the local machine. In order to reset the state of all migrations (for example, to be able to remove old migration files), see the [Hasura documentation](https://hasura.io/docs/latest/migrations-metadata-seeds/resetting-migrations-metadata/). + +## Arguments + +
+ For Aerie deployments before v2.8.0 + +`aerie_db_migration` takes the following command line arguments: + +```sh +aerie_db_migration.py [-h] (-a | -r) [--all] [-db DB_NAMES [DB_NAMES ...]] [-p HASURA_PATH] [-e ENV_PATH] [-n NETWORK_LOCATION] +``` + +| Option Name | Option Description | +|--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| +| `-h`, `--help` | Shows the help message. | +| `-a`, `--apply` | Apply migration steps to specified databases. | +| `-r`, `--revert` | Revert migration steps to specified databases. | +| `--all` | Apply (Revert) _ALL_ unapplied (applied) migration steps to specified databases. | +| `-db DB_NAMES [DB_NAMES ...]`,
`--db-names DB_NAMES [DB_NAMES ...]` | List of databases to migrate. Migrates all available databases if unspecified. | +| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` for Aerie. Defaults to `./hasura` | +| `-e ENV_PATH`, `--env-path ENV_PATH` | The path to the `.env` file used to deploy Aerie. **Must** define `AERIE_USERNAME` and `AERIE_PASSWORD`. Defaults to `.env`. | +| `-n NETWORK_LOCATION`, `--network-location NETWORK_LOCATION` | The network location of the database. Defaults to `localhost`. | + +It is necessary to at least specify `--apply` or `--revert`. + +If you are not running `aerie_db_migration` from within `deployment`, or if the `hasura` directory containing the `config.yaml` and the `migrations` directory is not in the directory that `aerie_db_migration` is being run from, then it is necessary to specify `--hasura-path`. +
+ +`aerie_db_migration.py` takes the following command line arguments: + +```sh +aerie_db_migration.py [-h] (-a | -r) [--all] [-p HASURA_PATH] [-e ENV_PATH] [-n NETWORK_LOCATION] +``` + +| Option Name | Option Description | +|--------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| +| `-h`, `--help` | Shows the help message. | +| `-a`, `--apply` | Apply migration steps to the database. | +| `-r`, `--revert` | Revert migration steps to the database. | +| `--all` | Apply (Revert) _ALL_ unapplied (applied) migration steps to the database. | +| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` for Aerie. Defaults to `./hasura` | +| `-e ENV_PATH`, `--env-path ENV_PATH` | The path to the `.env` file used to deploy Aerie. **Must** define `AERIE_USERNAME` and `AERIE_PASSWORD`. Defaults to `.env`. | +| `-n NETWORK_LOCATION`, `--network-location NETWORK_LOCATION` | The network location of the database. Defaults to `localhost`. | + +It is necessary to at least specify `--apply` or `--revert`. + +If you are not running `aerie_db_migration.py` from within `deployment`, +or if the `hasura` directory containing the `config.yaml` and the `migrations` directory is not in the directory +that `aerie_db_migration.py` is being run from, then it is necessary to specify `--hasura-path`. + +## Migrating a Database + +
+ For Aerie deployments before v2.8.0 + +If `--all` has been specified, then `aerie_db_migration.py` will automatically apply or revert all applicable changes to the specified databases (or all available databases if none are specified). It will then output the details of each change applied, followed by the overall status. For example: + +```sh +> aerie_db_migration.py -a -db AerieMerlin --all +########### +AerieMerlin +########### +VERSION TYPE NAME +1667319761264 up test_migration +INFO migrations applied +INFO Metadata reloaded +INFO Metadata is consistent + +############### +Database Status +############### + +Database: AerieMerlin +VERSION NAME SOURCE STATUS DATABASE STATUS +1667319761264 test_migration Present Present +``` + +Otherwise, `aerie_db_migration` will enter a step-by-step mode, where you first select which database to migrate, and then select whether +to apply each available migration step. For example: + +```sh +> aerie_db_migration.py -a -db AerieMerlin +############################### +AERIE DATABASE MIGRATION HELPER +############################### + +0) Quit the migration helper + +1) Database: AerieMerlin +VERSION NAME SOURCE STATUS DATABASE STATUS +1667319761264 test_migration Present Not Present + +Select a database to migrate (0-1): 1 + +########### +AerieMerlin +########### + +MIGRATION STEPS AVAILABLE: +VERSION NAME SOURCE STATUS DATABASE STATUS +1667319761264 test_migration Present Not Present + +CURRENT STEP: + +VERSION TYPE NAME +1667319761264 up test_migration + +Apply 1667319761264_test_migration? (y/n): _ +``` + +Entering `y` will apply the migration and then proceed to the next step, +should one exist. Entering `n` will return to the Database Selection screen. + +Entering `q` at any point will exit the program. + +
+ +If `--all` has been specified, then `aerie_db_migration.py` will automatically apply or revert all applicable changes to the database. +It will then output the details of each change applied, followed by the overall status. For example: + +```sh +> aerie_db_migration.py -a --all +############################### +AERIE DATABASE MIGRATION HELPER +############################### +VERSION TYPE NAME +1 up sample_migration +INFO migrations applied +INFO Metadata reloaded +INFO Metadata is consistent + +############### +Database Status +############### +VERSION NAME SOURCE STATUS DATABASE STATUS +1 sample_migration Present Present +``` + +Otherwise, `aerie_db_migration` will enter a step-by-step mode, where you will be prompted +whether to apply each available migration step. For example: + +```sh +> aerie_db_migration.py -a +############################### +AERIE DATABASE MIGRATION HELPER +############################### + +MIGRATION STEPS AVAILABLE: +VERSION NAME SOURCE STATUS DATABASE STATUS +1 sample_migration Present Not Present + +CURRENT STEP: + +VERSION TYPE NAME +1 up sample_migration + +Apply 1_sample_migration? (y/n/quit): _ +``` + +Entering `y` will apply the migration and then proceed to the next step, should one exist. +Entering `n`, `q`, or `quit` will exit the program. diff --git a/docs/deployment/advanced-database-migrations.mdx b/docs/deployment/advanced-database-migrations.mdx index 2e002b7..45fd3f6 100644 --- a/docs/deployment/advanced-database-migrations.mdx +++ b/docs/deployment/advanced-database-migrations.mdx @@ -1,5 +1,10 @@ # Advanced - Database Migrations +:::info Note +For Aerie deployments PRIOR to v3.3.0, please reference [this page](../advanced-database-migrations-OLD) +for accurate information as to how the migration script works. +::: + To aid in migrating between database versions we provide a Python script called [aerie_db_migration.py](https://github.com/NASA-AMMOS/aerie/blob/develop/deployment/aerie_db_migration). In order to run it, the following software is required: - [Python 3.9](https://www.python.org/downloads/) or later @@ -23,129 +28,57 @@ Alternatively, you can add the script to your `PATH` and run it from anywhere wi export PATH="$HOME/path/to/deployment:$PATH" ``` -The script expects all migration files present on the server to also exist on the local machine. In order to reset the state of all migrations (for example, to be able to remove old migration files), see the [Hasura documentation](https://hasura.io/docs/latest/migrations-metadata-seeds/resetting-migrations-metadata/). +The script expects all migration files present on the server to also exist on the local machine. +In order to reset the state of all migrations (for example, to be able to remove old migration files), see the [Hasura documentation](https://hasura.io/docs/latest/migrations-metadata-seeds/resetting-migrations-metadata/). + +## Subcommands -## Arguments +`aerie_db_migration.py` contains two subcommands: `status` and `migrate`. -
- For Aerie deployments before v2.8.0 +### Status -`aerie_db_migration` takes the following command line arguments: +Displays the current migration status of the database. ```sh -aerie_db_migration.py [-h] (-a | -r) [--all] [-db DB_NAMES [DB_NAMES ...]] [-p HASURA_PATH] [-e ENV_PATH] [-n NETWORK_LOCATION] +aerie_db_migration.py status [-h] [-p HASURA_PATH] [-e ENV_PATH] [--endpoint ENDPOINT] [--admin-secret ADMIN_SECRET] ``` -| Option Name | Option Description | -|--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| -| `-h`, `--help` | Shows the help message. | -| `-a`, `--apply` | Apply migration steps to specified databases. | -| `-r`, `--revert` | Revert migration steps to specified databases. | -| `--all` | Apply (Revert) _ALL_ unapplied (applied) migration steps to specified databases. | -| `-db DB_NAMES [DB_NAMES ...]`,
`--db-names DB_NAMES [DB_NAMES ...]` | List of databases to migrate. Migrates all available databases if unspecified. | -| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` for Aerie. Defaults to `./hasura` | -| `-e ENV_PATH`, `--env-path ENV_PATH` | The path to the `.env` file used to deploy Aerie. **Must** define `AERIE_USERNAME` and `AERIE_PASSWORD`. Defaults to `.env`. | -| `-n NETWORK_LOCATION`, `--network-location NETWORK_LOCATION` | The network location of the database. Defaults to `localhost`. | +| Option Name | Option Description | +|-----------------------------------------------|----------------------------------------------------------------------------------------------------------------| +| `-h`, `--help` | Shows the help message. | +| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` and migrations folder for Aerie. Defaults to `./hasura` | +| `-e ENV_PATH`, `--env-path ENV_PATH` | Envfile to load envvars from. | +| `--endpoint ENDPOINT ` | The http(s) endpoint for the Hasura instance. | +| `--admin-secret ADMIN_SECRET` | The admin secret | -It is necessary to at least specify `--apply` or `--revert`. +### Migrate -If you are not running `aerie_db_migration` from within `deployment`, or if the `hasura` directory containing the `config.yaml` and the `migrations` directory is not in the directory that `aerie_db_migration` is being run from, then it is necessary to specify `--hasura-path`. -
- -`aerie_db_migration.py` takes the following command line arguments: +Migrates the database. ```sh -aerie_db_migration.py [-h] (-a | -r) [--all] [-p HASURA_PATH] [-e ENV_PATH] [-n NETWORK_LOCATION] +aerie_db_migration.py migrate [-h] [-p HASURA_PATH] [-e ENV_PATH] [--endpoint ENDPOINT] [--admin-secret ADMIN_SECRET] (-a | -r) [--all] ``` -| Option Name | Option Description | -|--------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| -| `-h`, `--help` | Shows the help message. | -| `-a`, `--apply` | Apply migration steps to the database. | -| `-r`, `--revert` | Revert migration steps to the database. | -| `--all` | Apply (Revert) _ALL_ unapplied (applied) migration steps to the database. | -| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` for Aerie. Defaults to `./hasura` | -| `-e ENV_PATH`, `--env-path ENV_PATH` | The path to the `.env` file used to deploy Aerie. **Must** define `AERIE_USERNAME` and `AERIE_PASSWORD`. Defaults to `.env`. | -| `-n NETWORK_LOCATION`, `--network-location NETWORK_LOCATION` | The network location of the database. Defaults to `localhost`. | +| Option Name | Option Description | +|-----------------------------------------------|----------------------------------------------------------------------------------------------------------------| +| `-h`, `--help` | Shows the help message. | +| `-a`, `--apply` | Apply migration steps to the database. | +| `-r`, `--revert` | Revert migration steps to the database. | +| `--all` | Apply (Revert) _ALL_ unapplied (applied) migration steps to the database. | +| `-p HASURA_PATH`, `--hasura-path HASURA_PATH` | The path to the directory containing the `config.yaml` and migrations folder for Aerie. Defaults to `./hasura` | +| `-e ENV_PATH`, `--env-path ENV_PATH` | Envfile to load envvars from. | +| `--endpoint ENDPOINT ` | The http(s) endpoint for the Hasura instance. | +| `--admin-secret ADMIN_SECRET` | The admin secret | It is necessary to at least specify `--apply` or `--revert`. -If you are not running `aerie_db_migration.py` from within `deployment`, -or if the `hasura` directory containing the `config.yaml` and the `migrations` directory is not in the directory -that `aerie_db_migration.py` is being run from, then it is necessary to specify `--hasura-path`. - ## Migrating a Database -
- For Aerie deployments before v2.8.0 - -If `--all` has been specified, then `aerie_db_migration.py` will automatically apply or revert all applicable changes to the specified databases (or all available databases if none are specified). It will then output the details of each change applied, followed by the overall status. For example: - -```sh -> aerie_db_migration.py -a -db AerieMerlin --all -########### -AerieMerlin -########### -VERSION TYPE NAME -1667319761264 up test_migration -INFO migrations applied -INFO Metadata reloaded -INFO Metadata is consistent - -############### -Database Status -############### - -Database: AerieMerlin -VERSION NAME SOURCE STATUS DATABASE STATUS -1667319761264 test_migration Present Present -``` - -Otherwise, `aerie_db_migration` will enter a step-by-step mode, where you first select which database to migrate, and then select whether -to apply each available migration step. For example: - -```sh -> aerie_db_migration.py -a -db AerieMerlin -############################### -AERIE DATABASE MIGRATION HELPER -############################### - -0) Quit the migration helper - -1) Database: AerieMerlin -VERSION NAME SOURCE STATUS DATABASE STATUS -1667319761264 test_migration Present Not Present - -Select a database to migrate (0-1): 1 - -########### -AerieMerlin -########### - -MIGRATION STEPS AVAILABLE: -VERSION NAME SOURCE STATUS DATABASE STATUS -1667319761264 test_migration Present Not Present - -CURRENT STEP: - -VERSION TYPE NAME -1667319761264 up test_migration - -Apply 1667319761264_test_migration? (y/n): _ -``` - -Entering `y` will apply the migration and then proceed to the next step, -should one exist. Entering `n` will return to the Database Selection screen. - -Entering `q` at any point will exit the program. - -
- If `--all` has been specified, then `aerie_db_migration.py` will automatically apply or revert all applicable changes to the database. It will then output the details of each change applied, followed by the overall status. For example: ```sh -> aerie_db_migration.py -a --all +> aerie_db_migration.py migrate -a --all ############################### AERIE DATABASE MIGRATION HELPER ############################### @@ -166,7 +99,7 @@ Otherwise, `aerie_db_migration` will enter a step-by-step mode, where you will b whether to apply each available migration step. For example: ```sh -> aerie_db_migration.py -a +> aerie_db_migration.py migrate -a ############################### AERIE DATABASE MIGRATION HELPER ###############################