Merge branch 'develop' into feature/2709-ssp-section-1-validation

raft-tech · Nov 7, 2023 · a321529 · a321529
2 parents 96730a2 + fcd2a1d
commit a321529
Show file tree

Hide file tree

Showing 17 changed files with 320 additions and 114 deletions.
diff --git a/docs/Sprint-Review/sprint-83-summary.md b/docs/Sprint-Review/sprint-83-summary.md
@@ -0,0 +1,49 @@
+
+# Sprint 83 Summary
+
+09/30/23 - 10/11/23
+
+Velocity: Dev (18)
+
+## Sprint Goal
+* Complete parsing engine development for TANF Section (04) and begin SSP (01), close out subsmission history and metadata workflows (1613/12/10).
+* UX to continue regional staff and in-app messaging research, errors audit approach, and bridge onboarding to >95% of total users
+* DevOps to investigate singluar ClamAV (2429), resolve utlity images for CircleCI and evaluate CI/CD pipeline.
+
+
+## Tickets
+### Completed/Merged
+* [#1612 Detailed case level metadata](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1612)
+* [#1610 As a user, I need information about the acceptance of my data and a link for the error report](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1610)
+* [#1111 TANF (04) Parsing and Validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1111)
+
+### Ready to Merge
+* N/A
+
+### Submitted (QASP Review, OCIO Review)
+* N/A
+
+### Closed (not merged)
+* N/A
+
+## Moved to Next Sprint (Blocked, Raft Review, In Progress, Current Sprint Backlog)
+### In Progress
+* [#2536 [spike] Cat 4 validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2536)
+* [#2709 SSP (Section 1) validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2709)
+* [#2663 Investigate OWASP NightlyScan findings](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2663)
+
+### Blocked
+* N/A
+
+### Raft Review
+* [#2429 Singular ClamAV scanner](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2429)
+* [#2664 (bug) file extension](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2664)
+* [#2695 space-filled values update](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2695)
+* [#2411 As system admin, I need to view metadata on parsed datafiles](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2411)
+
+### Demo
+* Internal:
+    * 1111, 1610, 1612
+* External:
+    * 1111, 1610, 1612
+
diff --git a/docs/Sprint-Review/sprint-84-summary.md b/docs/Sprint-Review/sprint-84-summary.md
@@ -0,0 +1,61 @@
+# Sprint 84 Summary
+10/10/23 - 10/24/23
+
+Velocity: Dev (10)
+
+### Sprint Goal
+* Dev:
+    * Continue parsing engine development
+        * Complete SSP Sec (01) and SSP Sec (02)
+    * Resolve deployment blocker
+    * Coordinate w/ OFA and draft dev contingency plan for future gov shutdown
+* DevOps:
+    * 2429 - Singular Clam AV
+    * 2722 - Singular deployment workflow
+* UX: Resume regional staff research, synthesize in-app messaging research, continue supporting onboarding/utilization 
+* Prod: Find path forward on Sendgrid 
+
+## Tickets
+### Completed/Merged
+* [#2411 As system admin, I want to view metadata on parsed datafiles](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2411)
+* [#2429 Singular ClamAV Scanner](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2429)
+* [#2664 (bug) file extension](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2664)
+
+
+
+### Ready to Merge
+* [#2695 space-filled values update](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2695)
+* [#2725 file input render issue](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2725)
+
+
+### Submitted (QASP Review, OCIO Review)
+* [#2701 FETCH_STTS Infinite Request](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2701)
+* [#2709 SSP (Section 1) validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2709)
+
+### Closed (not merged)
+* N/A
+
+## Moved to Next Sprint (Blocked, Raft Review, In Progress, Current Sprint Backlog)
+### In Progress
+* [#2536 [spike] Cat 4 validation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2536)
+* [#1119 SSP Aggregate (03) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1119)
+* [#2592 Deploy celery as a separate cloud.gov app](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2592)
+* [#2599 Readability enhancements for error reports](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2599)
+* [#2683 ZAP result - CORS config issue](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2683)
+* [#2722 simplify workflows and de-bloat pipeline code](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2722)
+
+
+### Blocked
+* N/A
+
+### Raft Review
+* [#1118 SSP Closed Data (02) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1118)
+* [#1120 SSP Stratum (04) Parsing](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1120)
+* [#2116 Container Registry creation](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2116)
+* [Spike - Investigate OWASP nightly scan findings](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2663)
+
+### Demo
+* N/A
+
+
+
diff --git a/scripts/deploy-backend.sh b/scripts/deploy-backend.sh
@@ -51,6 +51,8 @@ set_cf_envs()
   "FRONTEND_BASE_URL"
   "LOGGING_LEVEL"
   "REDIS_URI"
+  "JWT_KEY"
+  "STAGING_JWT_KEY"
   )
 
   echo "Setting environment variables for $CGAPPNAME_BACKEND"
@@ -62,9 +64,13 @@ set_cf_envs()
         cf_cmd="cf unset-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
         $cf_cmd
         continue
+    elif [[ ("$var_name" =~ "STAGING_") && ("$CF_SPACE" = "tanf-staging") ]]; then
+        sed_var_name=$(echo "$var_name" | sed -e 's@STAGING_@@g')
+        cf_cmd="cf set-env $CGAPPNAME_BACKEND $sed_var_name ${!var_name}"
+    else
+      cf_cmd="cf set-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
     fi
-
-    cf_cmd="cf set-env $CGAPPNAME_BACKEND $var_name ${!var_name}"
+
     echo "Setting var : $var_name"
     $cf_cmd
   done
@@ -128,7 +134,7 @@ update_backend()
 bind_backend_to_services() {
     echo "Binding services to app: $CGAPPNAME_BACKEND"
 
-    if [ "$CFAPPNAME_BACKEND" = "tdp-backend-develop" ]; then
+    if [ "$CGAPPNAME_BACKEND" = "tdp-backend-develop" ]; then
       # TODO: this is technical debt, we should either make staging mimic tanf-dev 
       #       or make unique services for all apps but we have a services limit
       #       Introducing technical debt for release 3.0.0 specifically.

diff --git a/tdrs-backend/apt.yml b/tdrs-backend/apt.yml
@@ -2,7 +2,7 @@ cleancache: true
 keys:
   - https://www.postgresql.org/media/keys/ACCC4CF8.asc
 repos:
-  - deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main
+  - deb http://apt.postgresql.org/pub/repos/apt/ bookworm-pgdg main
 packages:
   - postgresql-client-12
   - libjemalloc-dev

diff --git a/tdrs-backend/tdpservice/parsers/fields.py b/tdrs-backend/tdpservice/parsers/fields.py
@@ -60,7 +60,8 @@ def parse_value(self, line):
 class TransformField(Field):
     """Represents a field that requires some transformation before serializing."""
 
-    def __init__(self, transform_func, item, name, type, startIndex, endIndex, required=True, validators=[], **kwargs):
+    def __init__(self, transform_func, item, name, type, startIndex, endIndex, required=True,
+                 validators=[], **kwargs):
         super().__init__(item, name, type, startIndex, endIndex, required, validators)
         self.transform_func = transform_func
         self.kwargs = kwargs

diff --git a/tdrs-backend/tdpservice/parsers/row_schema.py b/tdrs-backend/tdpservice/parsers/row_schema.py
@@ -114,7 +114,9 @@ def run_field_validators(self, instance, generate_error):
             else:
                 value = getattr(instance, field.name, None)
 
-            if field.required and not value_is_empty(value, field.endIndex-field.startIndex):
+            is_empty = value_is_empty(value, field.endIndex-field.startIndex)
+            should_validate = not field.required and not is_empty
+            if (field.required and not is_empty) or should_validate:
                 for validator in field.validators:
                     validator_is_valid, validator_error = validator(value)
                     is_valid = False if not validator_is_valid else is_valid

diff --git a/tdrs-backend/tdpservice/parsers/schema_defs/tanf/t1.py b/tdrs-backend/tdpservice/parsers/schema_defs/tanf/t1.py
@@ -92,7 +92,7 @@
                               validators.isNumber(),
                         ]),
                   Field(item="5", name='STRATUM', type='string', startIndex=22, endIndex=24,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isInStringRange(0, 99),
                         ]),
                   Field(item="7", name='ZIP_CODE', type='string', startIndex=24, endIndex=29,
@@ -128,15 +128,15 @@
                               validators.isInLimits(1, 2),
                         ]),
                   Field(item="15", name='RECEIVES_FOOD_STAMPS', type='number', startIndex=37, endIndex=38,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isInLimits(0, 2),
                         ]),
                   Field(item="16", name='AMT_FOOD_STAMP_ASSISTANCE', type='number', startIndex=38, endIndex=42,
                         required=True, validators=[
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="17", name='RECEIVES_SUB_CC', type='number', startIndex=42, endIndex=43,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isInLimits(0, 3),
                         ]),
                   Field(item="18", name='AMT_SUB_CC', type='number', startIndex=43, endIndex=47,
@@ -180,19 +180,19 @@
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="24A", name='TRANSITION_SERVICES_AMOUNT', type='number', startIndex=78, endIndex=82,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="24B", name='TRANSITION_NBR_MONTHS', type='number', startIndex=82, endIndex=85,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="25A", name='OTHER_AMOUNT', type='number', startIndex=85, endIndex=89,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="25B", name='OTHER_NBR_MONTHS', type='number', startIndex=89, endIndex=92,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.isLargerThanOrEqualTo(0),
                         ]),
                   Field(item="26AI", name='SANC_REDUCTION_AMT', type='number', startIndex=92, endIndex=96,
@@ -204,7 +204,7 @@
                               validators.oneOf([1, 2]),
                         ]),
                   Field(item="26AIII", name='FAMILY_SANC_ADULT', type='number', startIndex=97, endIndex=98,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.oneOf([0, 1, 2]),
                         ]),
                   Field(item="26AIV", name='SANC_TEEN_PARENT', type='number', startIndex=98, endIndex=99,
@@ -244,7 +244,7 @@
                               validators.oneOf([1, 2]),
                         ]),
                   Field(item="27", name='WAIVER_EVAL_CONTROL_GRPS', type='string', startIndex=113, endIndex=114,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.or_validators(validators.matches('9'), validators.isEmpty()),
                               validators.isAlphaNumeric(),
                         ]),
@@ -254,7 +254,7 @@
                                                 6, 7, 8, 9])
                         ]),
                   Field(item="29", name='FAMILY_NEW_CHILD', type='number', startIndex=116, endIndex=117,
-                        required=True, validators=[
+                        required=False, validators=[
                               validators.oneOf([1, 2]),
                         ]),
                   Field(item="-1", name='BLANK', type='string', startIndex=117, endIndex=156, required=False,