Merge branch 'develop' into 2799-catch-rpt-month-year-mismatches

.circleci/README.md

@@ -79,7 +79,7 @@ updated along with the secondary apps running in CF.
 
 ### Frontend/Backend
  - Before updating, make sure the current buildpacks that these apps use are supported by the latest OS. If they aren't you can update the manifest to point them to the correct buildpacks.
- - To update the apps you can either deploy each of the environments (sandbox, raft, qasp, etc) from CircleCi or you can use the `tdrs-deploy <ENVIRONMENT>` command from `commands.sh`. Assuming the buildpacks are up to date, that is all you need to do.
+ - To update the apps you can either deploy each of the environments (raft, qasp, etc) from CircleCi or you can use the `tdrs-deploy <ENVIRONMENT>` command from `commands.sh`. Assuming the buildpacks are up to date, that is all you need to do.
 
 ### Secondary apps
  - Before you can make the update, you need to ensure you have the CF plugin that allows you to do so. Download the binary for your respective OS [HERE](https://github.com/cloudfoundry/stack-auditor/releases) and follow the installation instructions [HERE](https://docs.cloudfoundry.org/adminguide/stack-auditor.html#install).

.circleci/deployment/commands.yml

@@ -79,19 +79,33 @@
       frontend-appname:
         default: tdp-frontend
         type: string
+      kibana-appname:
+        default: tdp-kibana
+        type: string
+      proxy-appname:
+        default: tdp-elastic-proxy
+        type: string
       cf-space:
         default: tanf-dev
         type: string
     steps:
       - get-app-deploy-strategy:
           appname: <<parameters.backend-appname>>
+      - run:
+          name: Install dependencies
+          command: |
+            sudo apt update
+            sudo add-apt-repository ppa:rmescandon/yq
+            sudo apt-get install yq
       - run:
           name: Deploy backend application
           command: |
             bash ./scripts/deploy-backend.sh \
               $DEPLOY_STRATEGY \
               <<parameters.frontend-appname>> \
               <<parameters.backend-appname>> \
+              <<parameters.kibana-appname>> \
+              <<parameters.proxy-appname>> \
               <<parameters.cf-space>>
 
   deploy-clamav:
@@ -115,6 +129,9 @@
       frontend-appname:
         default: tdp-frontend
         type: string
+      kibana-appname:
+        default: tdp-kibana
+        type: string
 # So the frontend knows what space its in for the banner.
 # I am unclear if the domain is a reliable metric to make this function
 # It seems like it might not be working
@@ -136,6 +153,7 @@
               $DEPLOY_STRATEGY \
               <<parameters.frontend-appname>> \
               <<parameters.backend-appname>> \
+              <<parameters.kibana-appname>> \
               <<parameters.cf-space>> \
               <<parameters.environment>>
 

.circleci/owasp/jobs.yml

@@ -56,6 +56,9 @@
       cf_space:
         type: string
         default: tanf-staging
+      cf_org:
+        type: string
+        default: "CF_ORG"
       target_env:
         type: enum
         enum: [ "staging", "develop", "prod" ]
@@ -64,6 +67,11 @@
       - sudo-check
       - cf-check
       - docker-compose-check
+      - login-cloud-dot-gov:
+          cf-password: <<parameters.cf_password>>
+          cf-space: <<parameters.cf_space>>
+          cf-org: <<parameters.cf_org>>
+          cf-username: <<parameters.cf_username>>
       - run-owasp-scan:
           environment: nightly
           target: backend
@@ -72,10 +80,6 @@
           environment: nightly
           target: frontend
           target_env: <<parameters.target_env>>
-      - login-cloud-dot-gov:
-          cf-password: <<parameters.cf_password>>
-          cf-space: <<parameters.cf_space>>
-          cf-username: <<parameters.cf_username>>
       - run:
           name: Run post-processing task to record OWASP ZAP results
           command: |

.gitconfig

@@ -7,6 +7,7 @@
 	allowed = [A-Z]+_KEY=..echo \".{S3_CREDENTIALS}\" [|] jq -r .+
 	allowed = ./tdrs-backend/.env.example:.*
 	allowed = ./tdrs-backend/docker-compose.yml:57:.*
+	allowed = ./tdrs-backend/manifest.proxy.yml:*
 	allowed = regexes.json:.*
 	allowed = ./scripts/copy-login-gov-keypair.sh:14:JWT_KEY=.*
 	allowed = scripts/deploy-backend.sh:.+:DJANGO_SECRET_KEY=..python -c .from secrets import token_urlsafe. print.token_urlsafe..*

docs/Sprint-Review/sprint-92-summary.md

@@ -0,0 +1,64 @@
+# Sprint 92 Summary
+
+01/31/2024 - 02/13/2024
+
+Velocity (Dev): 6
+
+## Sprint Goal
+* Dev:
+    * Continue parsing engine development and begin work on enhancement tickets
+        * #2536 Cat 4 validation 
+    * #1858 Secure OFA staff access to Kibana 
+        * Unblocks #1350 when complete 
+* DevOps:
+    * #2790 - Update deployment code to support Kibana and integrate with Standing Elastic instance
+* Design: 
+    * Tie up current documentation work
+    * Continue refinement of research roadmap
+
+
+## Tickets
+### Completed/Merged
+* [#2751 Resource Card updated with latest coding instructions](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2751)
+* [#1858 Spike: Secure Kibana access](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1858)
+* [#2781 As a developer,  I want to have documentation on django migration best practices](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2781)
+
+### Ready to Merge
+
+
+
+
+
+
+
+### Submitted (QASP Review, OCIO Review)
+* [#2790 Kibana Deployment](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2790)
+* [#2681 Section 1 Validation clean-up](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2681)
+
+
+
+### Closed (not merged)
+* N/A
+
+
+---
+
+## Moved to Next Sprint (In Progress, Blocked, Raft Review)
+### In Progress
+* [#1350 Kibana access from TDP](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1350)
+* [#2646 - Populate data file summary case aggregates differently per section](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2646)
+* [#2820 [bug] Uncaught exception re: parsing error preventing feedback report generation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2820)
+* [#2768 Fix production OWASP scan reporting](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2768)
+* [#2799 Generate error mismatching field rpt_month_year w/ header](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2799)
+
+
+
+### Blocked
+* N/A
+
+### Raft Review
+* [#2536 [spike] Cat 4 validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2536)
+* [#2592 Deploy celery as a separate cloud.gov app](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2592)
+* [#2746 As an STT, I need to know if there are issues with the DOBs reported in my data files](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2746)
+* [#2813 Reduce dev environment count](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2813)
+* [#2729 As a developer, I want to move migration commands in the pipeline to CircleCI](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2729)

docs/Technical-Documentation/Architecture-Decision-Record/008-deployment-flow.md

@@ -33,7 +33,6 @@ Within the dev space, there is no correlation for branch to environment as these
 
 | Dev Site | Frontend URL | Backend URL | Purpose                                          |
 | -------- | -------- | -------- |--------------------------------------------------|
-| Sandbox     | https://tdp-frontend-sandbox.app.cloud.gov | https://tdp-backend-sandbox.app.cloud.gov/admin/     | Space for development in a deployed environment |
 | A11y | https://tdp-frontend-a11y.app.cloud.gov | https://tdp-backend-a11y.app.cloud.gov/admin/ | Space for accessibility testing                  |
 | QASP | https://tdp-frontend-qasp.app.cloud.gov | https://tdp-backend-qasp.app.cloud.gov/admin/ | Space for QASP review                            |
 | raft | https://tdp-frontend-raft.app.cloud.gov | https://tdp-backend-raft.app.cloud.gov/admin/ | Space for Raft review                          |
@@ -53,4 +52,4 @@ Within the dev space, there is no correlation for branch to environment as these
 
 ## Notes
 
-- As of June 2022, CircleCI supplies environment variable key-value pairs to multiple environments (e.g. vendor's CircleCI deploys applications to dev and staging environments). The values from CircleCI are expected to be unique per environment, so until [#1826](https://github.com/raft-tech/TANF-app/issues/1826) is researched and addressed, these values will need to be manually corrected in cloud.gov immediately following the execution of the execution of the [`<env>-deployment` CircleCI workflow](../../.circleci/config.yml) CircleCI workflow. This workaround applies to backend applications in the TDP staging environment.
+- As of June 2022, CircleCI supplies environment variable key-value pairs to multiple environments (e.g. vendor's CircleCI deploys applications to dev and staging environments). The values from CircleCI are expected to be unique per environment, so until [#1826](https://github.com/raft-tech/TANF-app/issues/1826) is researched and addressed, these values will need to be manually corrected in cloud.gov immediately following the execution of the execution of the [`<env>-deployment` CircleCI workflow](../../.circleci/config.yml) CircleCI workflow. This workaround applies to backend applications in the TDP staging environment.

docs/Technical-Documentation/Failed-Deployment-Troubleshooting.md

@@ -13,6 +13,7 @@ In preparation for production-ready infrastructure, we wanted to create a living
 + [CircleCI failures](./Failed-Deployment-Troubleshooting.md#circleci-failures)
 + [Runtime failures](./Failed-Deployment-Troubleshooting.md#compilationruntime-failure)
 + [App Connectivity issues](./Failed-Deployment-Troubleshooting.md#app-connectivity-issues)
++ [App roll-back](./Failed-Deployment-Troubleshooting.md#revision-rollback)
 
 ## CircleCI failures
 **Symptom:** I deployed new code (via merging) but the app in Cloud.gov didn't update and is still running old code.
@@ -77,4 +78,26 @@ export [email protected]
 export LOGGING_LEVEL=DEBUG
 [...]
 bash scripts/deploy-backend.sh rebuild tdp-backend-raft tanf-dev
-```
+```
+
+## Revision Rollback
+
+First we need to get list of revisions and select a stable revision id.
+```cf revisions {app-name}```
+
+Then use the last successful guid, we can populate this reversion command:
+```
+cf curl v3/deployments \        
+-X POST \
+-d '{
+  "revision": {
+    "guid": "{last stable guid from list above}"
+  },
+  "relationships": {
+    "app": {
+      "data": {
+        "guid": "{current app guid}"
+      }
+    }
+  }
+}'```

docs/Technical-Documentation/TDP-environments-README.md

@@ -4,7 +4,6 @@
 
 | Dev Site | Frontend URL | Backend URL | Purpose |
 | -------- | -------- | -------- | -------- |
-| Sandbox     | https://tdp-frontend-sandbox.app.cloud.gov | https://tdp-frontend-sandbox.app.cloud.gov/admin/     | Space for devs to test in a deployed environment |
 | A11y | https://tdp-frontend-a11y.app.cloud.gov | https://tdp-frontend-a11y.app.cloud.gov/admin/ | Space for accessibility testing |
 | QASP | https://tdp-frontend-qasp.app.cloud.gov | https://tdp-frontend-qasp.app.cloud.gov/admin/ | Space for QASP review |
 | raft | https://tdp-frontend-raft.app.cloud.gov | https://tdp-frontend-raft.app.cloud.gov/admin/ | Space for raft review |