forked from HHS/TANF-app
-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'develop' into sprint-102-summary
- Loading branch information
Showing
59 changed files
with
2,854 additions
and
1,353 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # sprint-101-summary | ||
|
|
||
| 6/5/2024 - 6/18/2024 | ||
|
|
||
| **Dev:** | ||
|
|
||
| _**Prioritized DAC and Notifications Work**_  | ||
|
|
||
| * As sys admin, I want to be able to reparse datafile sets #2978 | ||
| * As a software engineer, I want to be able to test django-admin-508 #3008 | ||
| * As tech lead, I need the STT filter for search\_indexes to be updated #2950 | ||
| * As a data analyst I want to be notified of approaching data deadlines #2473 | ||
| * add `SENDGRID_API_KEY` to deploy.backend.sh #2677 | ||
| * Implement (small) data lifecycle (backup/archive ES) #3004 | ||
| * As a developer I want to test django-508 repo #2980\ | ||
|
|
||
|
|
||
| **DevOps:** | ||
|
|
||
| _**Successful deployments across environments and pipeline stability investments**_ | ||
|
|
||
| * Application health monitoring #831 | ||
|
|
||
| **Design:** | ||
|
|
||
| _**Close out error guide work, coordinate with dev on a plan for Cat 3 problems introduced by Cat 2 work, support spec-writing for upcoming work, and continued error audit dev ticket refinement.**_ | ||
|
|
||
| * Error Report Guide #2847 is going through final edits  | ||
| * Walk-on Dear Colleague letter link update to this PR (or spin up a separate ticket if deployment of the letter to OFA's website doesn't align to this) | ||
| * Deliver spec for #3014 (Blanked-out values in Submission History) | ||
| * \#3021 Updated KC Release Notes & Update Indicator FAQ - stretch goal for this sprint | ||
| * Write follow-on / spec tickets from #2909 findings - stretch/ongoing lift | ||
| * Category 3 error messages clean-up #2792 - stretch/ongoing lift | ||
|
|
||
| ## Tickets | ||
|
|
||
| ### Completed/Merged | ||
|
|
||
| * [#2980 As a developer I want to test django-508 repo](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2980) | ||
| * [#2892 Correct misleading error message for unaligned reporting year/q against header year/q](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2892) | ||
| * [#2909 \[Research Spike\] OOtB OFA Kibana Experience & DIGIT Data Access](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2909) | ||
| * [#2991 As tech lead, I need the sftp file transfer feature to be deprecated](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2991) | ||
| * [#2847 \[Design Deliverable\] Error Report Knowledge Center Explainer](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2847) | ||
| * [#3024 2897 follow-on for a11y-related enhancement ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3024) | ||
| * [#2897 As a data analyst I want finalized language and guidance resources in Submission History & Error Reports ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2897) | ||
|
|
||
| ### Submitted (QASP Review, OCIO Review) | ||
|
|
||
| * [#2133 \[Dev\] Enhancement for Request Access form (Tribe discoverability) ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2133) | ||
| * [#3023 as STT approved user, I need my IP address whitelisted so i can access TDP](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3023) | ||
| * [#3000 \[Design Deliverable\] TDP Poster for summer 2024 conferences](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3000) | ||
| * [#2795 As tech lead, I need TDP to detect duplicate records within a file and not store them in the db. ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2795) | ||
| * [#2693 \[Error Audit\] Category 2 error messages clean-up ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2693) | ||
| * [#2801 Friendly name cleanup ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2801) | ||
| * [#2883 Pre-Made Reporting Dashboards on Kibana](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2883) | ||
| * [#3021 \[Design Deliverable\] Updated KC Release Notes & Update Indicator FAQ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3021) | ||
| * [#2954 Extend SESSION\_COOKIE\_AGE](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2954) | ||
|
|
||
| ### Ready to Merge | ||
|
|
||
| * | ||
|
|
||
| ### Closed (Not Merged) | ||
|
|
||
| * [#2491 Create root-level docker-compose configuration file(s)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2491) | ||
| * [#1690 As a system admin, I need a way to be redirected to frontend from DAC](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1690) | ||
| * [#2351 As a user I want to be notified when the files are being scanned or uploaded when I push upload button](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2351) | ||
| * [#2591 Allow `manage.py` commands to be run by circleci](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2591) | ||
|
|
||
| ### Moved to Next Sprint  | ||
|
|
||
| **In Progress**  | ||
|
|
||
| * [#3004 Implement (small) data lifecycle (backup/archive ES)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3004) | ||
| * [#831 \[Spike\] As a Tech Lead, I want to get alerts when there is a backend or frontend error that affects an STT user ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/831) | ||
| * [#2978 As sys admin, I want to be able to reparse datafile sets](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2978) | ||
|
|
||
| #### Blocked | ||
|
|
||
| * | ||
|
|
||
| **Raft Review** | ||
|
|
||
| * [#2950 As tech lead, I need the STT filter for search\_indexes to be updated ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2950) | ||
| * [#3008 As a software engineer, I want to be able to test django-admin-508](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3008) | ||
| * [#3016 Spike - Cat2 Validator Improvement](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3016) | ||
| * [#2473 As a data analyst I want to be notified of approaching data deadlines](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2473) |
120 changes: 120 additions & 0 deletions
120
docs/Technical-Documentation/cloud-foundry-db-upgrade.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| # Cloud Foundry, Cloud.gov AWS RDS Database Upgrade | ||
|
|
||
| ## Process | ||
|
|
||
| If you are performing this process for the staging or production, you need to ensure you are performing the changes through the [HHS](https://github.com/HHS/TANF-app) repo and not the [Raft](https://github.com/raft-tech/TANF-app) repo. | ||
| <br/> | ||
|
|
||
| ### 1. SSH into a backend app in your desired environment | ||
| ```bash | ||
| cf ssh tdp-backend-<APP> | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 2. Create a backup of all the databases in the ENV's RDS instance | ||
| Note: you can get the required field values from `VCAP_SERVICES`. | ||
| ```bash | ||
| /home/vcap/deps/0/apt/usr/lib/postgresql/<CURRENT VERSION>/bin/pg_dump -h <HOST> -p <PORT> -d <DB_NAME> -U <DB_USER> -F c --no-acl --no-owner -f <FILE_NAME>.pg | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 3. Copy the backup(s) to your local machine | ||
| Note: This assumes you ran the backup command above in the home directory of the app. As an added bonus for later steps, you should execute this command from somewhere within `tdrs-backend` directory! Make sure not to commit the files/directories that are copied to your local directory. | ||
| ```bash | ||
| cf ssh tdp-backend--<APP> -c 'tar cfz - ~/app/*.pg' | tar xfz - -C . | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 4. Verify backup file size(s) match the backup size(s) in the app | ||
| ```bash | ||
| ls -lh <PWD>/home/vcap/app | ||
| ``` | ||
| As an added verification step, you should consider restoring the backups into a local server and verifying the contents with `psql` or `pgAdmin`. | ||
| <br/><br/> | ||
|
|
||
| ### 5. Update the `version` key in the `json_params` item in the `database` resource in the `main.tf` file in the environment(s) you're upgrading with the new database server version | ||
| ```yaml | ||
| json_params = "{\"version\": \"<NEW VERSION>\"}" | ||
| ``` | ||
| <br/> | ||
| ### 6. Update the `postgresql-client` version to the new version in `tdrs-backend/apt.yml` | ||
| ```yaml | ||
| - postgresql-client-<NEW VERSION> | ||
| ``` | ||
| Note: if the underlying OS for CloudFoundry is no longer `cflinuxfs4` you may also need to update the repo we point to for the postgres client binaries. | ||
| <br/><br/> | ||
|
|
||
| ### 7. Update the postgres container version in `tdrs-backend/docker-compose.yml` | ||
| ```yaml | ||
| postgres: | ||
| image: postgres:<NEW VERSION> | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 8. Update Terraform state to delete then re-create RDS instance | ||
| Follow the instuctions in the `terraform/README.md` and proceed from there. Modify the `main.tf` file in the `terraform/<ENV>` directory to inform TF of the changes. To delete the existing RDS instance you can simply comment out the whole database `resource` in the file (even though you made changes in the steps above). TF will see that the resource is no longer there, delete it, and appropriately update it's state. Then you simply re-comment the database `resource` back in with the changes you made in previous steps. TF will create the new RDS instance with your new updates, and also update the state in S3. | ||
| <br/><br/> | ||
|
|
||
| ### 9. Bind backend to the new RDS instance to get credentials | ||
| ```bash | ||
| cf bind-service tdp-backend-<APP> tdp-db-<ENV> | ||
| ``` | ||
| Be sure to re-stage the app when prompted | ||
| <br/><br/> | ||
|
|
||
| ### 10. Apply the backend manifest to begin the restore process | ||
| If you copied the backups as mentioned in the note from step 3, the backups will be copied for you to the app instance in the command below. If not, you will need to use `scp` to copy the backups to the app instance after running the command below. | ||
| ```bash | ||
| cf push tdp-backend-<APP> --no-route -f manifest.buildpack.yml -t 180 --strategy rolling | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 11. SSH into the app you just pushed | ||
| ```bash | ||
| cf ssh tdp-backend-<APP> | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 12. Create the appropriate database(s) in the new RDS server | ||
| Note: you can get the required field values from `VCAP_SERVICES`. | ||
| ```bash | ||
| /home/vcap/deps/0/apt/usr/lib/postgresql/<NEW VERSION>/bin/createdb -U <DB_USER> -h <HOST> <DB_NAME> | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 13. Restore the backup(s) to the appropriate database(s) | ||
| Note: you can get the required field values from `VCAP_SERVICES`. | ||
| ```bash | ||
| /home/vcap/deps/0/apt/usr/lib/postgresql/<NEW VERSION>/bin/pg_restore -p <PORT> -h <HOST> -U <DB_USER> -d <DB_NAME> <FILE_NAME>.pg | ||
| ``` | ||
| During this step, you may see errors similar to the message below. Note `<DB_USER>` is imputed in the message to avoid leaking environment specific usernames/roles. | ||
| ```bash | ||
| pg_restore: from TOC entry 215; 1259 17313 SEQUENCE users_user_user_permissions_id_seq <DB_USER> | ||
| pg_restore: error: could not execute query: ERROR: role "<DB_USER>" does not exist | ||
| Command was: ALTER TABLE public.users_user_user_permissions_id_seq OWNER TO <DB_USER>; | ||
| ``` | ||
| and the result and total amount of these errors should be: | ||
| ```bash | ||
| pg_restore: warning: errors ignored on restore: 68 | ||
| ``` | ||
| If this is what you see, everything is OK. This happens because the `pg_dump` doesn't remove owner associations on sequences for some reason. But you will see in the blocks above that `pg_restore` correctly alters the sequence owner to the new database user. | ||
| <br/><br/> | ||
|
|
||
| ### 14. Use `psql` to get into the database to check state | ||
| Note: you can get the required field values from `VCAP_SERVICES`. | ||
| ```bash | ||
| /home/vcap/deps/0/apt/usr/lib/postgresql/<NEW VERSION>/bin/psql <RDS URI> | ||
| ``` | ||
| <br/> | ||
|
|
||
| ### 15. Re-deploy or Re-stage the backend and frontend apps | ||
| Pending your environment you can do this GitHub labels or you can re-stage the apps from Cloud.gov. | ||
| <br/><br/> | ||
|
|
||
| ### 16. Access the re-deployed/re-staged apps and run a smoke test | ||
| - Log in | ||
| - Submit a few datafiles | ||
| - Make sure new and existing submission histories populate correctly | ||
| - Checkout the DACs data | ||
| <br/> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.