Merge branch 'develop' into 2722/circleci

commands.sh

@@ -227,7 +227,7 @@ tdrs-run-deploy() {
     --url https://circleci.com/api/v2/project/github/raft-tech/TANF-app/pipeline \
     --header 'Circle-Token: '$CIRCLE_CI_TOKEN \
     --header 'content-type: application/json' \
-    --data '{"parameters":{"run_dev_deployment": true, "target_env":"'$TARGET_ENV'"}, "branch":"'$BRANCH'"}'
+    --data '{"parameters":{"triggered": true, "run_dev_deployment": true, "target_env":"'$TARGET_ENV'"}, "branch":"'$BRANCH'"}'
 }
 
 # List all aliases and functions associated with tdrs

docs/Sprint-Review/sprint-83-summary.md

@@ -0,0 +1,49 @@
+
+# Sprint 83 Summary
+
+09/30/23 - 10/11/23
+
+Velocity: Dev (18)
+
+## Sprint Goal
+* Complete parsing engine development for TANF Section (04) and begin SSP (01), close out subsmission history and metadata workflows (1613/12/10).
+* UX to continue regional staff and in-app messaging research, errors audit approach, and bridge onboarding to >95% of total users
+* DevOps to investigate singluar ClamAV (2429), resolve utlity images for CircleCI and evaluate CI/CD pipeline.
+
+
+## Tickets
+### Completed/Merged
+* [#1612 Detailed case level metadata](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1612)
+* [#1610 As a user, I need information about the acceptance of my data and a link for the error report](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1610)
+* [#1111 TANF (04) Parsing and Validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1111)
+
+### Ready to Merge
+* N/A
+
+### Submitted (QASP Review, OCIO Review)
+* N/A
+
+### Closed (not merged)
+* N/A
+
+## Moved to Next Sprint (Blocked, Raft Review, In Progress, Current Sprint Backlog)
+### In Progress
+* [#2536 [spike] Cat 4 validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2536)
+* [#2709 SSP (Section 1) validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2709)
+* [#2663 Investigate OWASP NightlyScan findings](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2663)
+
+### Blocked
+* N/A
+
+### Raft Review
+* [#2429 Singular ClamAV scanner](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2429)
+* [#2664 (bug) file extension](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2664)
+* [#2695 space-filled values update](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2695)
+* [#2411 As system admin, I need to view metadata on parsed datafiles](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2411)
+
+### Demo
+* Internal:
+    * 1111, 1610, 1612
+* External:
+    * 1111, 1610, 1612
+

docs/Sprint-Review/sprint-84-summary.md

@@ -0,0 +1,61 @@
+# Sprint 84 Summary
+10/10/23 - 10/24/23
+
+Velocity: Dev (10)
+
+### Sprint Goal
+* Dev:
+    * Continue parsing engine development
+        * Complete SSP Sec (01) and SSP Sec (02)
+    * Resolve deployment blocker
+    * Coordinate w/ OFA and draft dev contingency plan for future gov shutdown
+* DevOps:
+    * 2429 - Singular Clam AV
+    * 2722 - Singular deployment workflow
+* UX: Resume regional staff research, synthesize in-app messaging research, continue supporting onboarding/utilization 
+* Prod: Find path forward on Sendgrid 
+
+## Tickets
+### Completed/Merged
+* [#2411 As system admin, I want to view metadata on parsed datafiles](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2411)
+* [#2429 Singular ClamAV Scanner](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2429)
+* [#2664 (bug) file extension](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2664)
+
+
+
+### Ready to Merge
+* [#2695 space-filled values update](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2695)
+* [#2725 file input render issue](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2725)
+
+
+### Submitted (QASP Review, OCIO Review)
+* [#2701 FETCH_STTS Infinite Request](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2701)
+* [#2709 SSP (Section 1) validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2709)
+
+### Closed (not merged)
+* N/A
+
+## Moved to Next Sprint (Blocked, Raft Review, In Progress, Current Sprint Backlog)
+### In Progress
+* [#2536 [spike] Cat 4 validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2536)
+* [#1119 SSP Aggregate (03) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1119)
+* [#2592 Deploy celery as a separate cloud.gov app](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2592)
+* [#2599 Readability enhancements for error reports](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2599)
+* [#2683 ZAP result - CORS config issue](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2683)
+* [#2722 simplify workflows and de-bloat pipeline code](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2722)
+
+
+### Blocked
+* N/A
+
+### Raft Review
+* [#1118 SSP Closed Data (02) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1118)
+* [#1120 SSP Stratum (04) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1120)
+* [#2116 Container Registry creation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2116)
+* [Spike - Investigate OWASP nightly scan findings](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2663)
+
+### Demo
+* N/A
+
+
+

scripts/deploy-backend.sh

@@ -51,6 +51,8 @@ set_cf_envs()
   "FRONTEND_BASE_URL"
   "LOGGING_LEVEL"
   "REDIS_URI"
+  "JWT_KEY"
+  "STAGING_JWT_KEY"
   )
 
   echo "Setting environment variables for $CGAPPNAME_BACKEND"
@@ -62,9 +64,13 @@ set_cf_envs()
         cf_cmd="cf unset-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
         $cf_cmd
         continue
+    elif [[ ("$var_name" =~ "STAGING_") && ("$CF_SPACE" = "tanf-staging") ]]; then
+        sed_var_name=$(echo "$var_name" | sed -e 's@STAGING_@@g')
+        cf_cmd="cf set-env $CGAPPNAME_BACKEND $sed_var_name ${!var_name}"
+    else
+      cf_cmd="cf set-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
     fi
-
-    cf_cmd="cf set-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
+
     echo "Setting var : $var_name"
     $cf_cmd
   done
@@ -128,7 +134,7 @@ update_backend()
 bind_backend_to_services() {
     echo "Binding services to app: $CGAPPNAME_BACKEND"
 
-    if [ "$CFAPPNAME_BACKEND" = "tdp-backend-develop" ]; then
+    if [ "$CGAPPNAME_BACKEND" = "tdp-backend-develop" ]; then
       # TODO: this is technical debt, we should either make staging mimic tanf-dev 
       #       or make unique services for all apps but we have a services limit
       #       Introducing technical debt for release 3.0.0 specifically.

tdrs-backend/apt.yml

@@ -2,7 +2,7 @@ cleancache: true
 keys:
   - https://www.postgresql.org/media/keys/ACCC4CF8.asc
 repos:
-  - deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main
+  - deb http://apt.postgresql.org/pub/repos/apt/ bookworm-pgdg main
 packages:
   - postgresql-client-12
   - libjemalloc-dev

tdrs-backend/tdpservice/parsers/fields.py

@@ -60,7 +60,8 @@ def parse_value(self, line):
 class TransformField(Field):
     """Represents a field that requires some transformation before serializing."""
 
-    def __init__(self, transform_func, item, name, type, startIndex, endIndex, required=True, validators=[], **kwargs):
+    def __init__(self, transform_func, item, name, type, startIndex, endIndex, required=True,
+                 validators=[], **kwargs):
         super().__init__(item, name, type, startIndex, endIndex, required, validators)
         self.transform_func = transform_func
         self.kwargs = kwargs

tdrs-backend/tdpservice/parsers/row_schema.py

@@ -114,7 +114,9 @@ def run_field_validators(self, instance, generate_error):
             else:
                 value = getattr(instance, field.name, None)
 
-            if field.required and not value_is_empty(value, field.endIndex-field.startIndex):
+            is_empty = value_is_empty(value, field.endIndex-field.startIndex)
+            should_validate = not field.required and not is_empty
+            if (field.required and not is_empty) or should_validate:
                 for validator in field.validators:
                     validator_is_valid, validator_error = validator(value)
                     is_valid = False if not validator_is_valid else is_valid